Data & Schemas Knowledgebase

Fuzzy Text Matching in PySpark UDF

String matching is a very common problem in business. You will learn how to perform address matching using FuzzyWuzzy in Incorta.

Screen Shot 2022-04-06 at 1.31.24 PM.png

Convert R DataFrame into Spark DataFrame

use as.DataFrame()and createDataFrame()to convert R data frame into Spark DataFrame.

Screen Shot 2022-04-05 at 7.13.05 PM.png

Convert Spark DataFrame into R DataFrame

Use collect() to collect the data from Spark DataFrame to R DataFrame.

Screen Shot 2022-04-05 at 5.49.04 PM.png

Interactive Analysis using Incorta Notebook and SparkR

Run SQL queries to do interactive analysis in Incorta Notebook with SparkR.

Screen Shot 2022-04-05 at 5.41.23 PM.png

How to handle Date Time in Spark MV

We would like to get the current hour, how can we get it in Spark SQL? We also need the time as of one hour ago, how do we do that?

Use SparkR DataFrame for Selecting, Filtering, and Grouping your Data

Use select(), filter(), arrange(), and group_by() functions to handle Spark DataFrame on R.

Screen Shot 2022-04-06 at 10.13.31 AM.png

Screen Shot 2022-04-06 at 10.13.54 AM.png

Get Basic Info about your DataFrame in SparkR

Get Basic Information about DataFrame in SparkR

Screen Shot 2022-04-05 at 5.26.16 PM.png

Screen Shot 2022-04-05 at 5.24.08 PM.png

Getting Started with R in Incorta Notebook

Overview R is a popular language in Data Science. Incorta supports creating materialized views (MV) with R. Your first Spark Program The minimum requirement for creating an MV includes the following: # Read data from Incorta into Spark df <- read(...

Use SparkSQL with SparkR

Overview In this recipe, you’ll learn how to run SQL queries from SparkR. After using the read() Incorta Notebook extension function to load data in Incorta Notebook, the data is in a SparkR DataFrame. You can use sql() to write SQL queries after cr...

Screen Shot 2022-04-05 at 4.00.41 PM.png

Creating Snapshot tables with Incorta

A snapshot table holds the same transactional data as its source system, with additional fields for tracking the snapshot date.

ML with IRIS data set using Incorta

This article shows how to use a classification Machine Learning (ML) algorithm on the Iris Flower data set. It first creates a model based on a training data set and then scores the the test data set and gives the prediction of the flower based on th...

How to connect to Incorta from external sql query tools like DbVisualizer or PSequel

Use external sql query tools like DbVisualizer or PSequel to connect to Incorta and run postgres sqls against Incorta data.

How to output a csv file for schema.table.column details

This python script takes an unzipped tenant export file or an unzipped schema export file and produces a csv file which has all the table column details like datatype, etc... in a schema. This csv file can then be imported into Incorta and a dashboar...