Python and PySpark libraries

In the materialized view, I'm attempting to import the python and pyspark libraries. But I get the error as in the image[native library can not be loaded ]. I have to use pyspark to conduct ML. 


Is there a way to install packages that are dependent?

Note: my sample code:

import pyspark.sql.functions as F
import numpy as np
from pyspark.ml.feature import Imputer
from pyspark.sql.functions import when
raw_data = read("Test_Schema.Diabetes")
import numpy as np
str1 = """SELECT * FROM raw_data"""
df1 = spark.sql(str1) 

2replies Oldest first
  • Oldest first
  • Newest first
  • Active threads
  • Popular
  • Hi Manikandan you can safely ignore the warning you have highlighted. Your script seems fine (although you import numpy twice which you should remove). If you are missing any Python packages, you can install them on your server with pip (or pip3 for Python 3). E.g. when I tested your script I got a warning I was missing module "numpy" so I ran "pip3 install numpy" from the command line.

    Like 1
    • Dustin Basil  Thanks. Now it is working. 

Like1 Follow
  • Status Answered
  • 11 mths agoLast active
  • 2Replies
  • 135Views
  • 2 Following