Running shell script with Oozie - oozie

Question

I am trying to run a sh script through Oozie, but I am facing a problem:
Cannot run program "script.sh" (in directory
"/mapred/local/taskTracker/dell/jobcache/job_201312061003_0001/attempt_201312061003_0001_m_000000_0/work"):
java.io.IOException: error=2, No such file or directory.
Please help me with necessary steps.

Answer 1

This error is really ambiguous. Here are some issues that have helped me to solve this issue.
-If you are running oozie workflows on a kerberized cluster, make sure to authenticate by passing your Kerberos Keytab as a argument:
...
<shell>
<exec>scriptPath.sh</exec>
<file>scriptPath.sh</file>
<file>yourKeytabFilePath</file>
</shell>
...
-In your shell File (scriptPath.sh), make sure ro remove first line shell reference.
#!usr/bin/bash
indeed, if this shell reference isn't deployed on all data nodes, this can lead to this error code.

Answer 2

I had this same issue because of something really silly. I added a shell block in the workflow, then I selected the corresponding sendMail.sh, but I forgot to add the file sendMail.sh in FILE +.

Answer 3

An Oozie shell action is executed on a random Hadoop node, i.e. not locally on the machine where the Oozie server is running. As Oleksii says, you have to make sure that your script is on the node that executes the job.
See the following complete examples of executing a shell action and an ssh action:
https://github.com/airawat/OozieSamples/tree/master/oozieProject/workflowShellAction
https://github.com/airawat/OozieSamples/tree/master/oozieProject/workflowSshAction

Answer 4

workflow.xml :
...
<shell>
<exec>script.sh</exec>
<file>scripts/script.sh</file>
</shell>
...
Make sure you have scripts/script.sh in the same folder in hdfs.

Answer 5

In addition to what others said, this can also be caused by a shell script having wrong line endings (e.g. CRLF for Windows). At least this happened to me :)

Answer 6

Try to give full path for HDFS like
<exec>/user/nathalok/run.sh</exec>
<file>/user/nathalok/run.sh#run.sh</file>
and ensure that in job.properties the path is mentioned correctly for the library and workflow.xml
oozie.libpath=hdfs://server/user/oozie/share/lib/lib_20150312161328/oozie
oozie.wf.application.path=hdfs://bcarddev/user/budaledi/Teradata_Flow

Answer 7

if your shell file exist in your project correlated dir. then it's your shell file format cause this error. you need to convert the format from dos to linux using dos2linux:dos2linux xxxx.sh

Answer 8

Also make sure that the shell scripts are UNIX compliant. If these shell scripts were written in windows environment then it appends windows specific end of lines (EOL) and these scripts are not recognized by the oozie. So you will get "no such file or directory found" in oozie shell actions.

Answer 9

workflow.xml would look something like this
<workflow-app name="HiveQuery_execution" xmlns="uri:oozie:workflow:0.5">
<start to="shell-3c43"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="shell-3c43">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<exec>/user/path/hivequery.sh</exec>
<file>/user/path/hivequery.sh#hivequery.sh</file>
<capture-output/>
</shell>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
Job.properties
jobTracker=xxxx.xxx.xxx.com:port
nameNode=hdfs://xxxx.xxx.xxx.com:port
better configure through UI, as suggested above