11-07-2023 08:05 AM
Is there any way to get the Spark execution plan in an MV when selecting Spark SQL? I know I can take the below approach, but I prefer to do this in Spark SQL and not have to rebuild the Spark SQL queries.
%pyspark
df = read("EBS_PARTY_COMMON.HZ_PARTIES")
df.registerTempTable("TBL")
spark.sql("""
SELECT count(*)
FROM TBL
""").explain(mode="formatted")
11-13-2023 01:10 PM
<IncortaCluster>/applications will get you to the SPARK screen which is full of drillable links and contains all kinds of information including ( I think ) the explain plan if you drill far enough.