I am pretty new to Unix environment.
I am trying to schedule two tasks in Unix server. The second task is dependent on the result of the first task. So, I want to run the first task. If there is no error then I want the second task to run automatically. But if the first task fails, I want to reschedule the first task again after 30 minutes.
I have no idea where to start from.
You don't need cron for this. A simple shell script is all you need:
#!/bin/sh
while :; do # Loop until the break statement is hit
if task1; then # If task1 is successful
task2 # then run task2
break # and we're done.
else # otherwise task1 failed
sleep 1800 # and we wait 30min
fi # repeat
done
Note that task1 must indicate success with an exit status of 0, and failure with nonzero.
As Wumpus sharply observes, this can be simplified to
#!/bin/sh
until task1; do
sleep 1800
done
task2
Related
I am using EMR cluster to run some job to run in parallel. Both of these job run in same cluster. I have put action_on_failure field to 'CONTINUE' so that if 1 task fails, the other should run in the cluster. I want end task which is EMRTerminateCluster to run after both these tasks gets completeted regardless of success or failure.
task2
task1 >> >> task4
task3
I want my dags to run in such a way that task4 only starts after task 2 and task3.
is there any way to this?
In my Autosys box scheduled to run every week, I have 2 jobs:
Job1 - Call up a shell script to generate a file
Job2 - Call up a
shell script to transfer the generated file
What happened is that for Job2, even though the file has been successfully transferred, there is no exit code from the shell script. This resulted in Job2 and the box being in RUNNING state and prevent the box from running at the next week schedule.
The ideal way is to amend the transfer shell script (in Job2) to return a proper exit code. But I do not have access to the shell script to make any change.
In JIL, is it possible to achieve either one of the following:
immediate after Job2 CMD execution, mark Job2 as success, OR
after X minutes of Job2 CMD execution, mark Job2 as success
Adding the term_run_time attribute to the JIL of Job2 will terminate the job if it has been running for more than the number of minutes specified.
For example, term_run_time: 60 sets a 60 minute termination timer.
I need to run multiple scripts in parallel. The scripts needs to ensure that x scripts are always running. I trigger 6 jobs in parallel, as soon as one finishes it triggers the next one, but id something fails it should stop processing any further jobs. I was able to get hold of a script which helped me in first part, but not sure how to handle the error tracking. I do not have GNU parallel
PROCS=6
SLEEP=20
while read LINE
do
while true
do
NUM=$(jobs | wc -l)
echo NUM=$NUM
if [ $NUM -lt $PROCS ]
then
stuff "$LINE" &
break
else
sleep $SLEEP
fi
done
done < textfile
wait
So above ensures 6 processes are always running. I want a way to find out if one of the process is failed, then do not trigger anymore jobs.
We have two Informatica jobs that run in parallel.
One starts at 11.40 CET and it has around 300 Informatica workflows in it out of which one is fact_sales.
The other job runs at 3.40 CET and it has around 115 workflows in it many of which are dependent on fact_sales in term of data consistency.
The problem is fact_sales should finish before certain workflows in process 2 starts for data to be accurate, but this doesnt happen generally.
What we are trying to do is to split the process 2 in such a way that fact_sales dependent workflows run only after the fact_sales has finished.
Can you provide me a way to go about writing a unix shell script that check the status of this fact_sales and if it successfull then kicks off other dependent workflows and if not then it should send a failure mail.
thanks
I don't see the need to write a custom shell script for this. Most of this is pretty standard/common functionality that can be implemented using Command Task and event waits.
**Process1 - runs at 11:50**
....workflow
...
fact_sales workflow. **Add a command task at the end
**that drops a flag, say, fact_sales_0430.done
...
....workflow..500
And all the dependent processes will have an event wait that waits on this .done file. Since there are multiple dependant workflows, make sure none of them deletes the file right away. You can drop this .done file at the end of the day or when the load starts for the next day.
workflow1
.....
dependantworkflow1 -- Event wait, waiting on fact_sales_0430.done (do not delete file).
dependantworkflow2 -- Event wait, waiting on fact_sales_0430.done (do not delete file).
someOtherWorkflow
dependantworkflow3 -- Event wait, waiting on fact_sales_0430.done (do not delete file).
....
......
A second approach can be as follows -
You must be running some kind of scheduler for launching these workflows.. since Informatica cant schedule multiple workflows in a set, it can only handle worklet/sessions at that level of dependency mgmt.
From the scheduler, create a dependency across the sales fact load wf and the other dependent workflows..
I think below mentioned script will work for you. Please udpate the parameters.
WAIT_LOOP=1
while [ ${WAIT_LOOP} -eq 1 ]
do
WF_STATUS=`pmcmd getworkflowdetails -sv $INFA_INTEGRATION_SERVICE -d $INFA_DOMAIN -uv INFA_USER_NAME -pv INFA_PASSWORD -usd Client -f $FOLDER_NAME $WORKFLOW_NAME(fact_sales) | grep "Workflow run status:" | cut -d'[' -f2 | cut -d']' -f1`
echo ${WF_STATUS} | tee -a $LOG_FILE_NAME
case "${WF_STATUS}" in
Aborted)
WAIT_LOOP=0
;;
Disabled)
WAIT_LOOP=0
;;
Failed)
WAIT_LOOP=0
;;
Scheduled)
WAIT_LOOP=0
;;
Stopped)
WAIT_LOOP=0
;;
Succeeded)
WAIT_LOOP=0
;;
Suspended)
WAIT_LOOP=0
;;
Terminated)
WAIT_LOOP=0
;;
Unscheduled)
WAIT_LOOP=0
;;
esac
if [ ${WAIT_LOOP} -eq 1 ]
then
sleep $WAIT_SECONDS
fi
done
if [ ${WF_STATUS} == "Succeeded" ]
then
pmcmd startworkflow -sv $INFA_INTEGRATION_SERVICE -d $INFA_DOMAIN -uv INFA_USER_NAME -pv INFA_PASSWORD -usd Client -f $FOLDER_NAME -paramfile $PARAMETER_FILE $WORKFLOW_NAME(dependent_one) | tee $LOG_FILE_NAME
else
(echo "Please find attached Logs for Run" ; uuencode $LOG_FILE_NAME $LOG_FILE_NAME )| mailx -s "Execution logs" $EMAIL_LIST
exit 1
fi
I can see you have main challenge - keep dependency between large number of infa workflows.
You have two options-
You can use some automated scheduling tool to set the dependency and run them one by one properly. There are many free tool but depending on your comfort/time/cost etc. you should choose. link here.
Secondly you can create your custom job-scheduler. I did a similar scheduler using UNIX script, oracle table. So here are steps for that -
Categorize all your workflows into groups. independent flow should go to group 1 and dependent flows on group 1 goes to group2 and so on.
Set your process to pick up one by one from above groups and kick them off. If kick off queue is empty then it should wait. call it loop2.
Keep a polling loop that will check status of kicked off flows. If failed, aborted etc. fail the process, mail to user and mark all 'in-queue/dependent' flows to failed. If running keep on polling. If succeeded give control to loop 2.
-if kick off queue is empty then go to next group only if all workflow in that group succeeded.
This is a bit tricky process but it paid off once you set it up. You can add as many workflows as you want and your maintenance will be much more smoother compared to infa scheduler or infa worklet etc.
You can fire a query from repository database using tables such REP_SESS_LOG and check if the status of the fact sales has succeeded or not. Then only you can proceed with the second job.
I would like to have a cron job which executes 3 shell scripts consecutively i.e., execution of next shell script depending on the completion of previous scripts.
How can I do it?
Here is an example showing a cron which executes 3 scripts at 9am Mon-Fri.
00 09 * * 1-5 script1.sh && script2.sh && script3.sh 2>&1 >> /var/tmp/cron.log
If any one of the scripts fails, the next script in the sequence will not be executed.
Write one script which calls these three scripts and pit it into cron.
To elaborate on yi_H's answer: You can combine them in one shell script in different ways, depending on what you want.
job1.sh
job2.sh
job3.sh
will run all three consecutively, regardless of the result.
job1.sh && job2.sh && job3.sh
will run all three, but it will stop if one of them fails (that is, if job1 returns an error, job2 and job3 will not run).