Oozie coordinator get day of the week - oozie

I am trying to create a condition in my Oozie workflow, where an action should be executed only on mondays (at the end of the workflow).
So far I added a decision node in the workflow, and the current date as parameter in the coordinator, and I need to test the day of the week.
coordinator.xml
<coordinator-app name="${project}_coord" frequency="${coord_frequency}" start="${coord_start_date}" end="${coord_end_date}" timezone="UTC" xmlns="uri:oozie:coordinator:0.1">
<controls>
<concurrency>1</concurrency>
<execution>LAST_ONLY</execution>
</controls>
<action>
<workflow>
<app-path>${wf_application_path}</app-path>
<configuration>
<property>
<name>currentDate</name>
<value>${coord:formatTime(coord:dateOffset(coord:nominalTime(), 0, ‘DAY’), “yyyyMMdd”)}</value>
</property>
</configuration>
</workflow>
</action>
workflow.xml
<decision name = "send_on_monday">
<switch>
<case to = "send_file">
${currentDay} eq "MON" <-------- test on day of the week
</case>
<default to = "sendSuccessEmail" />
</switch>
</decision>
<action name="send_file">
<ssh xmlns="uri:oozie:ssh-action:0.1">
<host>${remoteNode}</host>
<command>/pythonvenv</command>
<args>${fsProjectDir}/send_file.py</args>
</ssh>
<ok to="sendSuccessEmail"/>
<error to="sendTechnicalFailureEmail"/>
</action>
I didn't find information on how to get the day of the week with EL functions.
Any help is appreciated.

I found a solution by using wf:actionData in a decision node :
workflow.sh
<action name="getDayOfWeek">
<ssh xmlns="uri:oozie:ssh-action:0.1">
<host>${remoteNode}</host>
<command>${fsProjectDir}/scripts/dayOfWeek.sh</command>
<capture-output/>
</ssh>
<ok to="send_on_monday"/>
<error to="sendTechnicalFailureEmail"/>
</action>
<decision name="send_on_monday">
<switch>
<case to = "send_file">
${wf:actionData('getDayOfWeek')['day'] eq 'Mon'}
</case>
<default to = "sendSuccessEmail" />
</switch>
</decision>
dayOfWeek.sh
#!/bin/sh
DOW=$(date +"%a")
echo day=$DOW

Related

Oozie variable for all actions

Is there a way to set up a global variable for all actions in the workflow?
I need to define variable containing a value and then the same variable will be modified in the actions.
I tried:
<workflow-app name="test" xmlns="uri:oozie:workflow:0.5">
<global>
<configuration>
<property>
<name>variable1</name>
<value>/some/path</value>
</property>
</configuration>
</global>
.....
<action name="wf1">
....
<property>
<name>variable1</name>
<value>/some/other/path</value>
</property>
</action>
....
<action name="wf2">
.....
<property>
<name>variable1</name>
<value>/some/second/path</value>
</property>
....
</action>
<action name="createFolder">
<fs>
<mkdir path="${variable1}"/>
</fs>
<ok to="End"/>
<error to="Kill"/>
</action>
I would like to let actions to modified the value and then use it in another action. Is it possible? Right now I´m getting VARIABLE variable1 cannot be resolved
You can do this using action configuration.
You can even define default values for each action type.

Oozie workflow not running properly

I have created a new Oozie workflow in Hue UI based on the below hql query.
things.hql
drop table output_table;
create table output_table like things;
insert overwrite table output_table select t.* from things_dup td right outer join things t on (td.item_id = t.item_id) where td.item_id is null;
insert overwrite table things_dup select * from things;
The tables are,
things table
item_id product
1 soap
2 chocklate
things_dup
item_id product
1 soap
when i run the hql seperately
hadoop dfs -f things.hql
its working fine. things_dup table have updated properly.
But when i run the workflow, things_dup table have not updated.
insert overwrite table things_dup select * from things;
Can any one know why? Please help me to fix this issue.
Workflow.xml
<workflow-app name="Things_workflow" xmlns="uri:oozie:workflow:0.4">
<start to="Things_workflow"/>
<action name="Things_workflow">
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>/user/cloudera/hive-site.xml</job-xml>
<script>things.hql</script>
<file>/user/cloudera/hive-site.xml#hive-site.xml</file>
</hive>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
action
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>localhost.localdomain:8021</job-tracker>
<name-node>hdfs://localhost.localdomain:8020</name-node>
<job-xml>/user/cloudera/hive-site.xml</job-xml>
<script>things.hql</script>
<file>/user/cloudera/hive-site.xml#hive-site.xml</file>
</hive>
Thanks,
manimekalai
I also got issue with hive oozie but finally solved.
Please match you workflow.xml with this one.
Please must put this job-xml line before configuration node.
<?xml version="1.0" encoding="UTF-8"?>
<workflow-app xmlns="uri:oozie:workflow:0.2" name="pig-hive-wf">
<start to="hive-node" />
<action name="hive-node">
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>hive-site.xml</job-xml>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
<property>
<name>oozie.hive.defaults</name>
<value>/user/cloudera/oozie/pig_hive_data_WF/hive-site.xml</value>
</property>
</configuration>
<script>/user/cloudera/oozie/pig_hive_data_WF/load_data.q</script>
<param>LOCATION=/user/${wf:user()}/oozie/pig_hive_data_WF/output/pig_loaded_data</param>
<file>hive-conf.xml#hive-conf.xml</file>
</hive>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Pig Hive job is failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end" />
</workflow-app>
you can copy hive-site.xml file from /etc/hive/conf/hive-site.xml and put within workflow dir.

Pass -Dpig.notification.listener argument to Pig action in Oozie (Oozie client build version: 3.3.2-cdh4.7.1)

I want pass -Dpig.notification.listener argument to Pig action in oozie. But when I am doing this the pig is failing and there is no clear error description. Below is the Pig action snippet. Any suggestion how to pass listener to pig action.
<action name="FinalReport">
<pig>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/home/hadoop/surjan/path" />
</prepare> <script>${nameNode}//home/hadoop/surjan/wf/pigs/FinalReport.pig</script>
<param>inDate=20160430</param>
</pig>
<ok to="success" />
<error to="kill" />
</action>
EDIT: I found the solution for passing the -D arguments. Just in case if anybody is facing same issue.
<action name="FinalReport">
<pig>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/home/hadoop/surjan/path" />
</prepare>
<configuration>
<property>
<name>pig.notification.listener</name>
<value>com.surjan.util.counter.MyListener</value>
</property>
</configuration>
<script>${nameNode}//home/hadoop/surjan/wf/pigs/FinalReport.pig</script>
<param>inDate=20160430</param>
<argument>Dpig.notification.listener=com.surjan.util.counter.CounterListener</argument>
<archive>archive/mycounter.jar#mycounter</archive>
</pig>
<ok to="success" />
<error to="kill" />
enter code here

Oozie Invalid Transition

<workflow-app name="Oozie_app" xmlns="uri:oozie:workflow:0.1">
<start to="TransformWeatherData"/>
<action name="TransformWeatherData">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>default</value>
</property>
</configuration>
<exec>/home/kingsly/working_directory/copyFromLocal.sh</exec>
<file>/home/kingsly/working_directory/copyFromLocal.sh</file>
</shell>
<ok to="Oozie_app"/>
<error to="end"/>
</action>
<end name='end' />
I am new to Oozie and i have created a workflow and job.properties file
This is how my workflow.xml looks
When i submit this workflow i am getting error as
Error: E0708 : E0708: Invalid transition, node [TransformWeatherData] transition [Oozie_app]
please help me resolve this .
My main objective is to move a file from local machine to HDfs and i have included the Hadoop command in the shell script
You were referring to a missing node. I fixed this:
<workflow-app name="Oozie_app" xmlns="uri:oozie:workflow:0.1">
<start to="TransformWeatherData"/>
<action name="TransformWeatherData">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>default</value>
</property>
</configuration>
<exec>/home/kingsly/working_directory/copyFromLocal.sh</exec>
<file>/home/kingsly/working_directory/copyFromLocal.sh</file>
</shell>
<ok to="end"/>
<error to="kill" />
</action>
<kill name="kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name='end' />

How to replace attributes in XML transforms without the name attribute

I am using SlowCheetah to transform my Log4Net files when I publish. However, it can't seem to distinguish between the attributes in different appender sections.
My Log4Net.config looks basically like this:
<?xml version="1.0" encoding="utf-8" ?>
<log4net>
<appender name="SmtpAppender" type="log4net.Appender.SmtpAppender">
<to value="DevEmail" />
<from value="DevEmail" />
<subject value="Dev Warning" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="Time: %date%newlineHost: %property{log4net:HostName}%newlineClass: %logger%newlineUser: %property{user}%newlineMessage: %message%newline%newline%newline" />
</layout>
<threshold value="WARN" />
</appender>
<appender name="FatalSmtpAppender" type="log4net.Appender.SmtpAppender">
<to value="DevEmail" />
<from value="DevEmail" />
<subject value="Dev Fatal" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="Time: %date%newlineHost: %property{log4net:HostName}%newlineClass: %logger%newlineUser: %property{user}%newlineMessage: %message%newline%newline%newline" />
</layout>
<threshold value="FATAL" />
</appender>
</log4net>
And my transform file looks like this:
<?xml version="1.0" encoding="utf-8" ?>
<log4net xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<appender name="SmtpAppender" type="log4net.Appender.SmtpAppender">
<to value="ProductionEmail" xdt:Transform="SetAttributes" />
<from value="ProductionEmail" xdt:Transform="SetAttributes" />
<subject value="Production Warning" xdt:Transform="SetAttributes" />
</appender>
<appender name="FatalSmtpAppender" type="log4net.Appender.SmtpAppender">
<to value="ProductionEmail" xdt:Transform="SetAttributes" />
<from value="ProductionEmail" xdt:Transform="SetAttributes" />
<subject value="Production Fatal" xdt:Transform="SetAttributes" />
</appender>
</log4net>
The problem is that the transformed config has the same subject attribute value for both appenders; I guess when it hits the SetAttributes it can't tell which tag it's looking for, so it transforms all of them. What it the correct syntax to tell it to only find the elements within the same appender? I assume I need to use the xdt:Locator attribute, but I can't do Match(name) like I do for web.config because these elements don't have a name attribute. The appender element has a name attribute, but I don't know how to tell it to match based on the parent element's name.
I know that I could use replace on the appender node, with the match(Name), but then I would be replacing the entire node, including a bunch of elements such as the layout which I don't want to be transformed (and thus have multiple copy-pastes of the same code, which I would like to avoid).
I found the answer in this MSDN article: http://msdn.microsoft.com/en-us/library/dd465326.aspx.
I needed to use xdt:Locator="Match(name)" on the parent <appender> node, and then xdt:Transform on the child nodes. I had tried this previously but had used xdt:locator="Match(name)" instead of xdt:Locator="Match(name)"... The attribute is case sensitive.

Resources