Save and Reuse ML Models
I have built an MV with spark python that creates and tests an ML predictive model. Then it runs a batch of data against the prediction and saves the result. Currently it does this all in one MV. How would I go about creating and saving the model in one place (MV?) and then running the data against it in another MV? Can models be saved and loaded to file system (with pickle perhaps) so that the model could be rebuilt on one schedule and the predictions run on another schedule?
Hi, Steve. I have an example using ML predictive model. Here is my posting. https://suziepyspark.blogspot.com/2020/08/using-incorta-and-pyspark-linear.html
To save the model you can use this function.
This will be store under the IncortaNode folder.
ls */* Ecommerce_Customer_001/data: part-00000-5bc587a0-64cb-4930-b340-6b5e50970e23-c000.snappy.parquet Ecommerce_Customer_001/metadata: part-00000 _SUCCESS
You can then load the model in another MV.
lr_model = LinearRegressionModel.load("models/Ecommerce_Customer_001")
Hope this can help you. Thanks!