Query physical schema data from PySpark using the incorta_sql() function

[dylanwan]

Introduction

The incorta_sql() function lets you directly query physical schema data in a PySpark MV without going through the old and cumbersome way.

What you need to know before reading this article

Previously, you could use spark.sql() to query the data from physical schema data by creating temp views. 

Let's review the previously tedious steps: 

Go to Schema -> Explore Data -> SQL generation -> Copy the Reference SQL to Notebook -> Load data and create temp view -> Copy and edit the Reference SQL in spark.sql() function 

Incorta Version

Incorta 2022.10.0 or later

Let's Go

df = incorta_sql("""
SELECT
    *
FROM
    SCHEMA_NAME.TABLE_NAME
""")

Spoiler

Please note that you can directly use schema table and join among them without registering them as Spark temp views.

Example: 

Related Material

• Query business schema data from PySpark using the incorta_sql_pg() function
• Run SQL Queries from PySpark and Incorta Notebook