Convert R DataFrame into Spark DataFrame

[suxinji]

Overview

In this recipe, you'll learn how to convert R DataFrame into Spark DataFrame. You can use as.DataFrame() or createDataFrame()to convert R data frame into Spark DataFrame. 

Solution

We assume that you would like to use R to transform the data and the R DataFrame df_r_select was already created:

The df_spark is now a Spark DataFrame that can be saved as a MV.

# convert R data frame into a Spark data frame
df_spark <- as.DataFrame(df_r_select)
# convert R data frame into a Spark data frame - method 2
df_spark <- createDataFrame(df_r_select)