Data & Schema Discussions

Resolved! Trying To Fit A Linear Regression with pyspark.ml

I am trying to fit a simple linear regression with pyspark.ml in a materialized viewThe packages I am using are as follows: from pyspark.ml.feature import VectorAssembler from pyspark.ml.regression import LinearRegression my code looks like this: #...

Resolved! Java version for data agent?

Cloud documentation for data agent says to use OpenJDK11 ( snip below ) but the linked page doesn't have an OpenJDK11. Numerically it looks like JDK19 is the most recent GA release ( there's no "Open" on the link ) -- is that good for the data ag...

Spark Sql Execution Plan

Is there any way to get the Spark execution plan in an MV when selecting Spark SQL? I know I can take the below approach, but I prefer to do this in Spark SQL and not have to rebuild the Spark SQL queries. %pysparkdf = read("EBS_PARTY_COMMON.HZ_PARTI...

Data source in Incorta

HII need assistance in setting up a data source in Incorta. I have a colleague who updates an Excel file monthly on his device. I would like to know the best way to use this file as a data source in Incorta, allowing my colleague to continue updating...

Resolved! PySpark log stdout

I am building materialized view with Python Spark and since I am very new to Python I am struggling a little bit. In order to do some debugging I would like to be able to log some hints from the code, most likely in stdout (but I will be happy with a...