hadoop streaming failed with error code 1 in RHadoop - r

Question

I am working with RHadoop by the following code:
Sys.setenv(HADOOP_OPTS="-Djava.library.path=/usr/local/hadoop/lib/native")
Sys.setenv(HADOOP_HOME="/usr/local/hadoop")
Sys.setenv(HADOOP_CMD="/usr/local/hadoop/bin/hadoop")
Sys.setenv(HADOOP_STREAMING="/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-3.0.0.jar")
Sys.setenv(JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64")
library(rJava)
library(rhdfs)
library(rmr2)
hdfs.init()
mapper = function (., X) {
n=nrow(X);
ones=matrix(rep(1,n),nrow=n,ncol=1);
ag=aggregate(cbind(ones,X[,1:79]),by=list(X[,80]),FUN="sum")
key=factor(ag[,1]);
keyval(key,split(ag[,-1],key))
}
reducer = function(k, A) {
keyval(k,list(Reduce('+', A)))
}
GroupSums <- from.dfs( mapreduce(input = "/ISCXFlowMeter.csv", map = mapper, reduce = reducer, combine = T))
When I run this code, I get an error as:
packageJobJar: [/tmp/hadoop-unjar7138506441946536619/] []
/tmp/streamjob6099552934186757596.jar tmpDir=null 2018-06-12
22:40:04,651 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032 2018-06-12 22:40:04,945 INFO client.RMProxy: Connecting
to ResourceManager at /0.0.0.0:8032 2018-06-12 22:40:05,201 INFO
mapreduce.JobResourceUploader: Disabling Erasure Coding for path:
/tmp/hadoop-yarn/staging/uel/.staging/job_1528838017005_0012
2018-06-12 22:40:06,158 INFO mapred.FileInputFormat: Total input files
to process : 1 2018-06-12 22:40:06,171 INFO net.NetworkTopology:
Adding a new node: /default-rack/127.0.0.1:9866 2018-06-12
22:40:06,233 INFO mapreduce.JobSubmitter: number of splits:2
2018-06-12 22:40:06,348 INFO Configuration.deprecation:
yarn.resourcemanager.system-metrics-publisher.enabled is deprecated.
Instead, use yarn.system-metrics-publisher.enabled 2018-06-12
22:40:06,608 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1528838017005_0012 2018-06-12 22:40:06,610 INFO
mapreduce.JobSubmitter: Executing with tokens: [] 2018-06-12
22:40:06,945 INFO conf.Configuration: resource-types.xml not found
2018-06-12 22:40:06,945 INFO resource.ResourceUtils: Unable to find
'resource-types.xml'. 2018-06-12 22:40:07,022 INFO
impl.YarnClientImpl: Submitted application
application_1528838017005_0012 2018-06-12 22:40:07,249 INFO
mapreduce.Job: The url to track the job:
http://uel-Deskop-VM:8088/proxy/application_1528838017005_0012/
2018-06-12 22:40:07,251 INFO mapreduce.Job: Running job:
job_1528838017005_0012 2018-06-12 22:40:09,301 INFO mapreduce.Job: Job
job_1528838017005_0012 running in uber mode : false 2018-06-12
22:40:09,305 INFO mapreduce.Job: map 0% reduce 0% 2018-06-12
22:40:09,337 INFO mapreduce.Job: Job job_1528838017005_0012 failed
with state FAILED due to: Application application_1528838017005_0012
failed 2 times due to AM Container for
appattempt_1528838017005_0012_000002 exited with exitCode: 127
Failing this attempt.Diagnostics: [2018-06-12 22:40:08.734]Exception
from container-launch. Container id:
container_1528838017005_0012_02_000001 Exit code: 127
[2018-06-12 22:40:08.736]Container exited with a non-zero exit code
127. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : /bin/bash: /bin/java: No such file or
directory
[2018-06-12 22:40:08.736]Container exited with a non-zero exit code
127. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : /bin/bash: /bin/java: No such file or
directory
For more detailed output, check the application tracking page:
http://uel-Deskop-VM:8088/cluster/app/application_1528838017005_0012
Then click on links to logs of each attempt. . Failing the
application. 2018-06-12 22:40:09,368 INFO mapreduce.Job: Counters: 0
2018-06-12 22:40:09,369 ERROR streaming.StreamJob: Job not successful!
Streaming Command Failed! Error in mr(map = map, reduce = reduce,
combine = combine, vectorized.reduce, : hadoop streaming failed
with error code 1
>
ISCXFlowMeter.csv file in hadoop is available here: https://www.dropbox.com/s/rbppzg6x2slzcjz/ISCXFlowMeter.csv?dl=1
Could you please guide me how to rectify this issue?

Answer 1

After a while, by adding the following properties into mapred-site.xml, I could rectify the error.
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
But, the issue now is that, the key-value is NULL after completing map-reduce. Any help, I appreciate it.