Getting Started with Spark SQL

[suxinji]

• Introduction
• What you need to know before reading this article
• Steps
• Navigation
• Incorta Notebook for SparkSQL
• Autocomplete for SparkSQL
• Use the DESC command to get the list of columns
• Use Incorta Notebook Visualization

 

Introduction

Incorta Materialized Views are created based on the parquet data from Incorta data storage. We can access the data loaded into Incorta using Spark SQL.

Incorta Notebook provides an interactive environment and can be used to perform ad hoc analysis.

What you need to know before reading this article

Incorta Spark Architecture

Concepts - Materialized View

Getting Started with Incorta Notebook

Steps

Navigation 

Go to MV -> Make sure Spark SQL is selected -> Click on Notebook

Incorta Notebook for SparkSQL

Incorta's notebook includes multiple paragraphs.  

When you are creating SparkSQL-based MV from Incorta Notebook, we assume that one of the paragraphs will be used. 

There are several features in the Incorta Notebook including shortcuts, markdown cells for documentation, and much more.

• Use Keyboard Shortcuts within an Incorta Notebook 
• Use Markdown in Incorta Notebook (Not published, check the publish title if same)

Autocomplete for SparkSQL

When you type the schema name partially in the FROM clause, Incorta prompts the schema names for you to select.

Use the DESC command to get the list of columns

DESC describes the table statement in Incorta Spark SQL. The metadata information includes column name, column type, and column comment. 

Syntax: 

DESC Schema_Name.Table_Name 

Here is an example: 

Use Incorta Notebook Visualization

You can visualize your data using built-in data visualization within Incorta Notebook by changing the icon to change the different charts.