How to unnest data and obtain the first element from an array in SparkR? - r

Question

I am new to SparkR and trying first steps of data preparation.The dataset is something of this kind. I was trying to subset and select significant columns. My question is how can I select a column from an array element. I was trying something like this, which allowed me to select columns by un-nesting data but couldn't unnest and flatten the array to get it's first element. Helpful Link
select.col <- SparkR::select(data,c("parsed.nid","parsed.status","parsed.sections.element[0].name"))

Answer 1

I myself found a way to resolve this issue.This can be done in two simple steps :-
First we need to use explode() in SparkR, to get all the contents in
the list from that column.
Next, we need to use windowPartitionBy() in SparkR to create a
partitions and then we can get anything we want based on our
requirements like row_number(),dense_rank(),rank() etc. Like here we want the first element of the list, so I have used row_number function.
Snippet :
data.select <- SparkR::select(data,c("parsed.nid","parsed.status","parsed.sections"))
names(data.select) <- c("nid","status","sections")
categories <- SparkR::select(data.select,data.select$nid,data.select$status,explode(data.select$sections))
ws <- SparkR::orderBy(SparkR::windowPartitionBy("nid","status","sections"),"nid")
data.final <- SparkR::mutate(categories,row_num = over(row_number(), ws))
##If we want to get the first element of the array.
data.final <- data.final[data.final$row_num==1,]
Please add your suggestions as well.