.png "Employee"){kind=icon}
Introduction
Incorta MV Notebooks offer an alternative way to edit Spark code used to generate materialized views (MVs). This article provides a couple of tips that allows you to save your work while doing development.
What you need to know before reading this article
Understanding Incorta Materialized Views and Notebooks is essential.
Let's Go
Save a Dummy Dataframe
To save an MV for the first time, Incorta requires a dataframe to define its structure (column names, data types). This structure can change as you refine the MV.
If your code is not ready, but you would like to save it, you can create a dummy dataframe and save the MV.
Here is a dummy dataframe that can be used to save an MV.
df_dummy = spark.createDataFrame([['x']], ['c'])
save(df_dummy)
The first argument in the createDataFrame function is the data to be saved. It assumes a two dimension array or a list of lists. The inner list is a row and the outer list is a set of rows.
I gave a single column row and the next argument is a list of column names. The data type of the columns can be determined by Spark based on the value passed in.
This code creates an MV with a single column named 'c' containing one row with the value 'x'.
Save the notebook to the desktop
Incorta Notebooks allow you to export and import code.
Regularly exporting your work as a backup is crucial, especially during significant edits. While Incorta saves schema versions, local backups provide an extra layer of security.
Use the export feature to save your notebook code to your computer for easy retrieval in case of timeouts, accidental browser closures, or logouts.
Save Script only without Validating the code
For unsaved progress, you can leverage Incorta's script-only saving feature. This saves your incomplete code without validating the MV, allowing you to resume development later.
Conclusion
Incorta MV Notebook offers a powerful development environment, but unlike some standalone notebooks, lacks automatic saving. By implementing the techniques covered in this article, you can ensure your work is never lost.
Related Material
