3replies
-
Hi, IncortaVi
You can do data profiling using Spark Python in Incorta.
Method 1:
Using df.describe()
This function can provide min, max, count, mean, stddev. But only for data types of string and number.
Method 2:
Calculate each metric ourselves.
For more detail please find blow link:
https://suziepyspark.blogspot.com/2021/02/how-to-do-data-profiling-in-incorta.html
Hope it can help you.