Showing results for 
Search instead for 
Did you mean: 

Materialized Views with Scala


I am trying to refactor some tables that were previously using pyspark to use scala. 

However whenever I try to read in my data I am met with an error that states "table or view not found"

My code looks something like this:


import org.apache.spark.sql.{SparkSession, DataFrame}

// Define your schema
val schema = "schema_name"
val table = "table_name"

// Read the table into a DataFrame
val df: DataFrame =
.format("parquet") // table format
.option("header", "true") // Add options as needed
.option("inferSchema", "true")
// .schema(schema) // Specify the schema here


Trying to troubleshoot if I need to switch directories or anything I tried running


// Show all schemas in the current database
spark.sql("SHOW DATABASES").show()


which tells me that I am in a database named "default"

Running the following


// Show all tables in the current database
spark.sql("SHOW TABLES").show()


returns an empty dataframe with 3 columns named database, tableName, and isTemporary.

All 3 fields are blank

How can I go about connecting scala to the correct schema?