Materialized Views are computed via Spark (on top the Incorta parquet files). To run them, developers would need to submit Spark jobs that would undertake the required logic. Python and SQL are widely used among BI developers, database admins and data scientists. It's much easier to write your job in those languages (as opposed to Java or Scala for example).
The workflow of a materialized view is simple:
- User/developer submits the MV script (whether Python or SQL)
- The script is encapsulated in an Spark job, and gets submitted to Spark (on load events)
- The result is written back to a parquet file stored inside Incorta tenant directory
- Now Incorta can treat the MV as a normal table
- Upon another load event (whether full or incremental), the appropriate script will be resubmitted to Spark