0

Save and Reuse ML Models

I have built an MV with spark python that creates and tests an ML predictive model. Then it runs a batch of data against the prediction and saves the result. Currently it does this all in one MV. How would I go about creating and saving the model in one place (MV?) and then running the data against it in another MV? Can models be saved and loaded to  file system (with pickle perhaps) so that the model could be rebuilt on one schedule and the predictions run on another schedule? 

2replies Oldest first
  • Oldest first
  • Newest first
  • Active threads
  • Popular
  • Hi, Steve. I have an example using ML predictive model. Here is my posting. https://suziepyspark.blogspot.com/2020/08/using-incorta-and-pyspark-linear.html

    To save the model you can use this function. 

    lr_model.write().overwrite().save("models/Ecommerce_Customer_001")
    

    This will be store under the IncortaNode folder.

    /incorta/IncortaAnalytics/IncortaNode/models

    ls */*
    Ecommerce_Customer_001/data:
    part-00000-5bc587a0-64cb-4930-b340-6b5e50970e23-c000.snappy.parquet
    
    Ecommerce_Customer_001/metadata:
    part-00000  _SUCCESS

    You can then load the model in another MV.

    lr_model = LinearRegressionModel.load("models/Ecommerce_Customer_001")
    

    Hope this can help you. Thanks!

    Like
  • Thank you so much! This is exactly what I was looking for. 

    Best wishes!!

    Like
Like Follow
  • 7 days agoLast active
  • 2Replies
  • 13Views
  • 2 Following

Product Announcement


We are happy to
announce Incorta 4.8 !!!