Read a csv file in sparkR where columns have spaces - r

Question

Normally, when we read a csv file in R, the spaces are automatically converted to '.'
> df <- read.csv("report.csv")
> str(df)
'data.frame': 598 obs. of 61 variables:
$ LR.Number
$ Vehicle.Number
However, when we read the same csv file in sparkR, the space remains intact and is not handled implicitly by spark
#To read a csv file
df <- read.df(sqlContext, path = "report.csv", source = "com.databricks.spark.csv", inferSchema = "true", header="true")
printSchema(df)
root
|-- LR Number: string (nullable = true)
|-- Vehicle Number: string (nullable = true)
Because of this, performing any activity with the column causes a lot of trouble and need to be call like this
head(select(df, df$`LR Number`))
How can I explicitly handle this? How can sparkR implicitly handle this.
I am using sparkR 1.5.0 version

Answer 1

As a work around you could use the following piece of psuedo code
colnames_df<-colnames(df)
colnames_df<-gsub(" ","_",colnames_df)
colnames(df)<-colnames_df
Another solution is to save file somewhere and read using read.df()

Answer 2

Following worked for me
df = collect(df)
colnames_df<-colnames(df)
colnames_df<-gsub(" ","_",colnames_df)
colnames(df)<-colnames_df
df <- createDataFrame(sqlContext, df)
printSchema(df)
Here we need to locally collect the data first, which will convert spark data frame to normal R data frame. I am sceptical whether this is a good solution as I don't want to call collect. However I investigated and found that even to use ggplot libraries we need to convert this into a local data frame