Materialized Views with Scala

mkrieger — Sun, 28 Jan 2024 22:55:38 GMT

I am trying to refactor some tables that were previously using pyspark to use scala.

However whenever I try to read in my data I am met with an error that states "table or view not found"

My code looks something like this:

import org.apache.spark.sql.{SparkSession, DataFrame} // Define your schema val schema = "schema_name" val table = "table_name" // Read the table into a DataFrame val df: DataFrame = spark.read .format("parquet") // table format .option("header", "true") // Add options as needed .option("inferSchema", "true") // .schema(schema) // Specify the schema here .table(s"$schema.$table")

Trying to troubleshoot if I need to switch directories or anything I tried running

// Show all schemas in the current database spark.sql("SHOW DATABASES").show()

which tells me that I am in a database named "default"

Running the following

// Show all tables in the current database spark.sql("SHOW TABLES").show()

returns an empty dataframe with 3 columns named database, tableName, and isTemporary.

All 3 fields are blank

How can I go about connecting scala to the correct schema?

topic Materialized Views with Scala in Data & Schema Discussions

Materialized Views with Scala