Using R on a Hadoop installed cluster - r

Question

I am a R user. I know very little about Linux command, PuTTY or Hadoop/Hive. So please correct me, if I am wrong.
I am now working with a team. They have a running Ubuntu system on a cluster. I can use PuTTY to access this Ubuntu system and access the data files by using code:
user$hadoop fs -ls /datafolder/
or by using hive:
user$hive
hive>use datafolder;
hive>show tables;
On the opposite, the team that I am working with knows very little about R, so they want me to do the R part. I have installed R on the cluster, and also installed rJava HRive and other packages in R. (I am not sure I did this correctly, but R seems running OK).
Now I can do some testing. I can run the following code on R # cluster:
user$R
>install.packages(c('Rcpp', 'RJSONIO', 'bitops', 'digest', 'functional', 'stringr', 'plyr', 'reshape2','caTools'))
>Sys.setenv(HADOOP_CMD="/opt/cloudera/bin/hadoop")
>Sys.setenv(HADOOP_HOME="/opt/cloudera/lib/hadoop")
>Sys.setenv(HADOOP_STREAMING="/opt/cloudera/lib/hadoop-mapreduce/hadoop-streaming.jar")
>library(rmr2)
>library(rhdfs)
>hdfs.init()
Testing:
>ints = to.dfs(1:10)
>calc = mapreduce(input = ints, map = function(k,v) cbind(v, v/2, 2*v))
>test <- from.dfs(calc)
>test
I can successfully load "test" back by using from.dfs. It seemed like I can save a dummy dataset to Hadoop, and can get it back from Hadoop successfully (correct?)
Now, my question is, how to let R import those datasets that I can see from
user$hadoop fs -ls /datafolder/
or
>hive use datafolder;

Answer 1

this is example of wordcount with load result back to R:
Sys.setenv(HADOOP_CMD="/usr/bin/hadoop")
Sys.setenv(HADOOP_STREAMING="/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming.jar")
Sys.setenv(JAVA_HOME="/usr/java/jdk1.7.0_55-cloudera")
Sys.setenv(HADOOP_COMMON_LIB_NATIVE_DIR="/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/hadoop/lib/native")
Sys.setenv(HADOOP_OPTS="-Djava.library.path=HADOOP_HOME/lib")
library(rhdfs)
hdfs.init()
library(rmr2)
## space and word delimiter
map <- function(k,lines) {
words.list <- strsplit(lines, '\\s')
words <- unlist(words.list)
return( keyval(words, 1) )
}
reduce <- function(word, counts) {
keyval(word, sum(counts))
}
wordcount <- function (input, output=NULL) {
mapreduce(input=input, output=output, input.format="text", map=map, reduce=reduce)
}
## variables
hdfs.root <- '/user/node'
hdfs.data <- file.path(hdfs.root, 'data')
hdfs.out <- file.path(hdfs.root, 'out')
## run mapreduce job
##out <- wordcount(hdfs.data, hdfs.out)
system.time(out <- wordcount(hdfs.data, hdfs.out))
## fetch results from HDFS
results <- from.dfs(out)
results.df <- as.data.frame(results, stringsAsFactors=F)
colnames(results.df) <- c('word', 'count')
##head(results.df)
## sorted output TOP10
head(results.df[order(-results.df$count),],10)