I'm writing a shell script that basically transfers data files through SFTP to a database server and then invokes a pl/sql procedure which loads the data from those files (external tables) into internal database tables.
I've been doing some research on effective exception handling in shell scripts and it appears the set -e option can be used to terminate a script with an error whenever any command in the script runs which returns a non-zero exitcode.
So, my plan is to have a script which contains all of the processing that needs to get done (SFTP, moving/deleting files, calling pl/sql procedure, etc...) and to include set -e at the top of the script. I also plan to redirect output to a log file in this script.
Then, I plan to have another script that calls the main processing script and then emails the log that gets produced with either a "Success" or "Failure" indicator in the subject of the email.
Are there any "gotchas" that any of you can foresee in this approach or does this seem reasonable?
Sounds reasonable.
One thing you may also do make it one command and less scripts:
someSFTPscipt &> somelogfile.txt; if [ $? -eq 0 ]; then echo "Success"; else echo "Failure"; fi
someSFTPscipt &> somelogfile.txt; redirects the output of the script to a logfile
if [ $? -eq 0 ]; then echo "Success"; else echo "Failure"; fi checks whether it succeeded (returned 0) or failed (any other non-zero value). Simply replace the echo with your mail commands.
Thanks for all of the feedback on this.
I ended up going with this "wrapper" shell that calls the main processing shell. A cron is going to launch this daily in my particular case.
Comments are certainly welcome if this can be improved.
#!/bin/sh
################################################################################
# Author : Zack Macomber #
# Date : 02/22/2012 #
# Description: Calls main_process.sh and emails results of the process. #
# Also appends to master log file #
################################################################################
# Flag any errors that occur during processing
set -e
# Set newly created files to "rw" for everyone
umask 111
#############
# VARIABLES #
#############
EMAIL_RECIPIENTS=my_email#some_domain.com
MAIN_DIR=/scripts/
#############
# FUNCTIONS #
#############
send_email()
{
uuencode results.log results.log | \
mailx -s "DATA_LOAD $1 - consult attached log for details" $EMAIL_RECIPIENTS
}
################
# MAIN PROCESS #
################
cd $MAIN_DIR
sh main_process.sh > results.log && send_email SUCCESS || send_email FAILURE
cat results.log >> pub_data_load.log
exit 0
Related
I have a Parent job in Rundeck and in steps I have multiple reference jobs (that are created using "Job Reference - Execute another Job for each Node") and internally each of these reference jobs has different tasks.
Is there as way I can get Steps status (Pass or Fail) of the Parent job's to a file?
The purpose of this is to generate a report and attach to a mail which will have the success or failure of each step.
As you may have noticed, Rundeck only takes the Parent Job execution on Job Reference Step workflows (this is by design), in this case, we will have to "play" a bit with Rundeck.
We need the executions of the child jobs, for that, we will have to make them run individually, capture the state of each of these executions and place it in a template Markdown file which can be sent in a notification as an email template.
To call each job we will create a "fake parent job" which runs each child jobs individually (via API using cURL) and save the result (using jq to extract the values) on an Markdown file in an inline-script step, this is the script:
# script that obtains individual child jobs and uses it to a mail notification
# https://stackoverflow.com/questions/64590927/how-to-get-steps-status-of-a-rundeck-job-to-a-file
#####################################################
# rundeck instance values
rdeck_server="your_rundeck_host" #rundeck hostname
rdeck_port="4440" # rundeck tcp port
rdeck_api="36" # rundeck api version
jobid="9c667cb5-7f93-4c01-99b0-3249e75887e4 1c861d46-17a3-45ee-954d-74c6d7b597a0 ae9e455e-ca7a-440e-9ab4-bfd25827b287" # space separated child jobs id's
token="bmqlGuhEcekSyNtWAwuizER4YoJBgZdI" # user token
#####################################################
# "prudential" time between actions (in seconds)
pt="2"
#####################################################
# clean the file
echo "" > myfile.md
#####################################################
# add a header to a file
mydate=$(date +'%m/%d/%Y')
echo "# $mydate report" >> myfile.md
#####################################################
# 1) run the job via API and store the execution ID.
# 2) takes the execution ID and store job status via API
for myid in $jobid # iterate over jobs id's
do
# sleep
sleep $pt
# save the execution id (to get the status later) on a variable named "execid"
execid=$(curl -s -H accept:application/json --location --request POST "http://$rdeck_server:$rdeck_port/api/$rdeck_api/job/$myid/run?authtoken=$token" | jq -r '.id')
# sleep
sleep $pt
# save the status on a variable named "status"
status=$(curl -s --location --request GET "http://$rdeck_server:$rdeck_port/api/$rdeck_api/execution/$execid/state?authtoken=$token" | jq -r '.executionState')
# put the status on a file
echo "* job $myid is $status" >> myfile.md
# rundeck friendly output message
echo "job $myid is done, status: $status"
done
And within a Job Definition it would look like this:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>eb4826bc-cc49-46b5-9aff-351afb529197</id>
<loglevel>INFO</loglevel>
<name>FakeParent</name>
<nodeFilterEditable>false</nodeFilterEditable>
<notification>
<onfailure>
<email attachType='file' recipients='it#example.net' subject='info' />
</onfailure>
<onsuccess>
<email attachType='file' recipients='it#example.net' subject='info' />
</onsuccess>
</notification>
<notifyAvgDurationThreshold />
<plugins />
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<exec>echo "starting..."</exec>
</command>
<command>
<fileExtension>.sh</fileExtension>
<script><![CDATA[# script that obtains individual child jobs and uses it to a mail notification
# https://stackoverflow.com/questions/64590927/how-to-get-steps-status-of-a-rundeck-job-to-a-file
#####################################################
# rundeck instance values
rdeck_server="your_rundeck_host" #rundeck hostname
rdeck_port="4440" # rundeck tcp port
rdeck_api="36" # rundeck api version
jobid="9c667cb5-7f93-4c01-99b0-3249e75887e4 1c861d46-17a3-45ee-954d-74c6d7b597a0 ae9e455e-ca7a-440e-9ab4-bfd25827b287" # space separated child jobs id's
token="bmqlGuhEcekSyNtWAwuizER4YoJBgZdI" # user token
#####################################################
# "prudential" time between actions (in seconds)
pt="2"
#####################################################
# clean the file
echo "" > myfile.md
#####################################################
# add a header to a file
mydate=$(date +'%m/%d/%Y')
echo "# $mydate report" >> myfile.md
#####################################################
# 1) run the job via API and store the execution ID.
# 2) takes the execution ID and store job status via API
for myid in $jobid # iterate over jobs id's
do
# sleep
sleep $pt
# save the execution id (to get the status later) on a variable named "execid"
execid=$(curl -s -H accept:application/json --location --request POST "http://$rdeck_server:$rdeck_port/api/$rdeck_api/job/$myid/run?authtoken=$token" | jq -r '.id')
# sleep
sleep $pt
# save the status on a variable named "status"
status=$(curl -s --location --request GET "http://$rdeck_server:$rdeck_port/api/$rdeck_api/execution/$execid/state?authtoken=$token" | jq -r '.executionState')
# put the status on a file
echo "* job $myid is $status" >> myfile.md
# rundeck friendly output message
echo "job $myid is done, status: $status"
done]]></script>
<scriptargs />
<scriptinterpreter>/bin/bash</scriptinterpreter>
</command>
<command>
<exec>echo "done!"</exec>
</command>
</sequence>
<uuid>eb4826bc-cc49-46b5-9aff-351afb529197</uuid>
</job>
</joblist>
If you check the "notifications" section you will notice that it will send an email if it executes correctly or if it fails. You need to configure Rundeck so that it can send emails. I leave the steps to configure it:
Stop your Rundeck service.
Add the e-mail configuration on the rundeck-config.properties file (usually at /etc/rundeck path):
# e-mail notification settings
grails.mail.host=your-smtp-host.com
grails.mail.port=25
grails.mail.username=your-username
grails.mail.password=yourpassword
More info here.
Add the template configuration, specific for your job, also on rundeck-config.properties file (check the line 3, is the file path generated by the job script):
# project and job specific
rundeck.mail.YOUR-PROJECT-NAME.YOUR-JOB-NAME.template.subject=your-subject-string
rundeck.mail.YOUR-PROJECT-NAME.YOUR-JOB-NAME.template.file=/path/to/your/myfile.md
rundeck.mail.YOUR-PROJECT-NAME.YOUR-JOB-NAME.template.log.formatted=true # (if true, prefix log lines with context information)
Start your rundeck service.
At the moment of executing your job, you will see the individual execution and the report in the inbox.
Need to check for distribution of a file in an array programmatically. Logging into a master server and then would like to check for file on workers using simple ssh. So far I have:
ssh $HOSTNAME "[ -e '$HOSTNAME:/directory/filename' ] && echo 'Exists'"
Based on some of the logging output, I know the ssh is successful, but how can I get the test to return a message to the master server? Running the above returns nothing.
SSH will exit with the same exit code as the command that you run on the remote host. If that command is a test, then the exit code will match what you would normally expect from a test.
I would suggest the following:
Simplify your command to only run the test over SSH
Run the echo on your local machine
It doesn't seem correct that you have $HOSTNAME: in front of your path.
ssh "$HOSTNAME" "test -e '/directory/filename'" && echo 'Exists'
I personally find if statements to be much more easily understandable, which is an optional change if you are willing to go that route:
if ssh "$HOSTNAME" "test -e '/directory/filename'"; then
echo "Exists"
else
echo "Does not exist" >&2
exit 1
fi
I am trying to get the status code out of an Rscript run in an non-interactive way in the form of a bash script. This step is part of larger data processing cycle that involves db2 scripts among other things.
So I have the following contents in a script sample.sh:
Rscript --verbose --no-restore --no-save /home/R/scripts/sample.r >> sample.rout
when this sample.sh is run it always returns a status code of 0, irrespective of if the sample.r script run fully or error out in an intermediate step.
I tried the following things but no luck
1 - in the sample.sh file, I added an if and else condition for a return code like the below, but it again wrote back 0 despite sample.r failing in one of the functions inside.
if Rscript --verbose --no-restore --no-save /home/R/scripts/sample.r >> sample.rout
then
echo -e "0"
else
echo -e "1"
fi
2 - I also tried a wrapper script, like in a sample.wrapper.sh file
r=0
a=$(./sample.sh)
r=$?
echo -e "\n return code of the script is: $a\n"
echo -e "\n The process completed with status: $r"
here also I did not get the expected '1' in the case of failure of the sample.r in an intermediate step on both the variables a and r. Ideally, i would like a way to capture the error (as '1') in a.
Could someone please advice how to get rscript to write '0' only in case of completion of the entire script without any errors and '1' in all other cases?
greatly appreciate the input! thank you!
I solved the problem by returning the status code in addition to echo. below is the code snipped from sample.sh script. In addition, in sample.R code i have added trycatch to catch the errors and quit(status = 1).
function fun {
if Rscript --verbose --no-restore --no-save /home/R/scripts/sample.r > sample.rout 2>&1
then
echo -e "0"
return 0
else
echo -e "1"
return 1
fi
}
fun
thanks everyone for your inputs.
The above code works for me. I modified it so that I could reuse the function and have it exit when there's an error
Rscript_with_status () {
rscript=$1
if Rscript --vanilla $rscript
then
return 0
else
exit 1
fi
}
run r scripts by:
Rscript_with_status /path/to/script/sample.r
Your remote script needs to provide a proper exit status.
You can make a 1st test by providing i.e. "exit 1" at the end of the remote script and see that it will make a difference.
remote.sh:
#!/bin/sh
exit 1
From local machine:
ssh -l username remoteip /home/username/remote.sh
echo $?
1
But the remote script should also provide to you the exit status of the last executed command. Experiment further by modifying your remote script:
#!/bin/sh
#exit 1
/bin/false
The exit status of the remote command will now also be 1.
There is a existing script which run daily. Is it possible to track return code of scripts after its completion to track that it was successfully finished or not in its previous run? I don't want to modify the existing script.
After the script was executed, its exit code is stored in the $? variable, which you could save to a file. For example
/path/to/script.sh
echo $? >> /path/to/script.log
You probably want to save the date too:
/path/to/script.sh
result=$?
echo $(date) $result >> /path/to/script.log
Here is the scenario,
$hostname
server1
I have the below script in server1,
#!/bin/ksh
echo "Enter server name:"
read server
rsh -n ${server} -l mquser "/opt/hd/ca/scripts/envscripts.ksh"
qdisplay
# script ends.
In above script I am logging into another server say server2 and executing the script "envscripts.ksh" which sets few alias(Alias "qdisplay") defined in it.
I can able to successfully login to server1 but unable to use the alias set by script "envscripts.ksh".
Geting below error,
-bash: qdisplay: command not found
can some please point out what needs to be corrected here.
Thanks,
Vignesh
The other responses and comments are correct. Your rsh command needs to execute both the ksh script and the subsequent command in the same invocation. However, I thought I'd offer an additional suggestion.
It appears that you are writing custom instrumentation for WebSphere MQ. Your approach is to remote shell to the WMQ server and execute a command to display queue attributes (probably depth).
The objective of writing your own instrumentation is admirable, however attempting to do it as remote shell is not an optimal approach. It requires you to maintain a library of scripts on each MQ server and in some cases to maintain these scripts in different languages.
I would suggest that a MUCH better approach is to use the MQSC client available in SupportPac MO72. This allows you to write the scripts once, and then execute them from a central server. Since the MQSC commands are all done via MQ client, the same script handles Windows, UNIX, Linux, iSeries, etc.
For example, you could write a script that remotely queried queue depths and printed a list of all queues with depth > 0. You could then either execute this script directly against a given queue manager or write a script to iterate through a list of queue managers and collect the same report for the entire network. Since the scripts are all running on the one central server, you do not have to worry about getting $PATH right, differences in commands like tr or grep, where ksh or perl are installed, etc., etc.
Ten years ago I wrote the scripts you are working on when my WMQ network was small. When the network got bigger, these platform differences ate me alive and I was unable to keep the automation up and running. When I switched to using WMQ client and had only one set of scripts I was able to keep it maintained with far less time and effort.
The following script assumes that the QMgr name is the same as the host name except in UPPER CASE. You could instead pass QMgr name, hostname, port and channel on the command line to make the script useful where QMgr names do not match the host name.
#!/usr/bin/perl -w
#-------------------------------------------------------------------------------
# mqsc.pl
#
# Wrapper for M072 SupportPac mqsc executable
# Supply parm file name on command line and host names via STDIN.
# Program attempts to connect to hostname on SYSTEM.AUTO.SVRCONN and port 1414
# redirecting parm file into mqsc.
#
# Intended usage is...
#
# mqsc.pl parmfile.mqsc
# host1
# host2
#
# -- or --
#
# mqsc.pl parmfile.mqsc < nodelist
#
# -- or --
#
# cat nodelist | mqsc.pl parmfile.mqsc
#
#-------------------------------------------------------------------------------
use strict;
$SIG{ALRM} = sub { die "timeout" };
$ENV{PATH} =~ s/:$//;
my $File = shift;
die "No mqsc parm file name supplied!" unless $File;
die "File '$File' does not exist!\n" unless -e $File;
while () {
my #Results;
chomp;
next if /^\s*[#*]/; # Allow comments using # or *
s/^\s+//; # Delete leading whitespace
s/\s+$//; # Delete trailing whitespace
# Do not accept hosts with embedded spaces in the name
die "ERROR: Invalid host name '$_'\n" if /\s/;
# Silently skip blank lines
next unless ($_);
my $QMgrName = uc($_);
#----------------------------------------------------------------------------
# Run the parm file in
eval {
alarm(10);
#Results = `mqsc -E -l -h $_ -p detmsg=1,prompt="",width=512 -c SYSTEM.AUTO.SVRCONN &1 | grep -v "^MQSC Ended"`;
};
if ($#) {
if ($# =~ /timeout/) {
print "Timed out connecting to $_\n";
} else {
print "Unexpected error connecting to $_: $!\n";
}
}
alarm(0);
if (#Results) {
print join("\t", #Results, "\n");
}
}
exit;
The parmfile.mqsc is any valid MQSC script. One that gathers all the queue depths looks like this:
DISPLAY QL(*) CURDEPTH
I think the real problem is that the r(o)sh cmd only executes the remote envscripts.ksh file and that your script is then trying to execute qdisplay on your local machine.
You need to 'glue' the two commands together so they are both executed remotely.
EDITED per comment from Gilles (He is correct)
rosh -n ${server} -l mquser ". /opt/hd/ca/scripts/envscripts.ksh ; qdisplay"
I hope this helps.
P.S. as you appear to be a new user, if you get an answer that helps you please remember to mark it as accepted, or give it a + (or -) as a useful answer