Checking ftp return codes from Unix script - unix

I am currently creating an overnight job that calls a Unix script which in turn creates and transfers a file using ftp. I would like to check all possible return codes. The man page for ftp doesn't list return codes. Does anyone know where to find a list? Anyone with experience with this? We have other scripts that grep for certain return strings in the log, and they send an email when in error. However, they often miss unanticipated codes.
I am then putting the reason into the log and the email.

The ftp command does not return anything other than zero on most implementations that I've come across.
It's much better to process the three digit codes in the log - and if you're sending a binary file, you can check that bytes sent was correct.
The three digit codes are called 'series codes' and a list can be found here

I wrote a script to transfer only one file at a time and in that script use grep to check for the 226 Transfer complete message. If it finds it, grep returns 0.
ftp -niv < "$2"_ftp.tmp | grep "^226 "

Install the ncftp package. It comes with ncftpget and ncftpput which will each attempt to upload/download a single file, and return with a descriptive error code if there is a problem. See the “Diagnostics” section of the man page.

I think it is easier to run the ftp and check the exit code of ftp if something gone wrong.
I did this like the example below:
# ...
ftp -i -n $HOST 2>&1 1> $FTPLOG << EOF
quote USER $USER
quote PASS $PASSWD
cd $RFOLDER
binary
put $FOLDER/$FILE.sql.Z $FILE.sql.Z
bye
EOF
# Check the ftp util exit code (0 is ok, every else means an error occurred!)
EXITFTP=$?
if test $EXITFTP -ne 0; then echo "$D ERROR FTP" >> $LOG; exit 3; fi
if (grep "^Not connected." $FTPLOG); then echo "$D ERROR FTP CONNECT" >> $LOG; fi
if (grep "No such file" $FTPLOG); then echo "$D ERROR FTP NO SUCH FILE" >> $LOG; fi
if (grep "access denied" $FTPLOG ); then echo "$D ERROR FTP ACCESS DENIED" >> $LOG; fi
if (grep "^Please login" $FTPLOG ); then echo "$D ERROR FTP LOGIN" >> $LOG; fi
Edit: To catch errors I grep the output of the ftp command. But it's truly it's not the best solution.
I don't know how familier you are with a Scriptlanguage like Perl, Python or Ruby. They all have a FTP module which you can be used. This enables you to check for errors after each command. Here is a example in Perl:
#!/usr/bin/perl -w
use Net::FTP;
$ftp = Net::FTP->new("example.net") or die "Cannot connect to example.net: $#";
$ftp->login("username", "password") or die "Cannot login ", $ftp->message;
$ftp->cwd("/pub") or die "Cannot change working directory ", $ftp->message;
$ftp->binary;
$ftp->put("foo.bar") or die "Failed to upload ", $ftp->message;
$ftp->quit;
For this logic to work user need to redirect STDERR as well from ftp command as below
ftp -i -n $HOST >$FTPLOG 2>&1 << EOF
Below command will always assign 0 (success) as because ftp command wont return success or failure. So user should not depend on it
EXITFTP=$?

lame answer I know, but how about getting the ftp sources and see for yourself

I like the solution from Anurag, for the bytes transfered problem I have extended the command with grep -v "bytes"
ie
grep "^530" ftp_out2.txt | grep -v "byte"
-instead of 530 you can use all the error codes as Anurag did.

You said you wanted to FTP the file there, but you didn't say whether or not regular BSD FTP client was the only way you wanted to get it there. BSD FTP doesn't give you a return code for error conditions necessitating all that parsing, but there are a whole series of other Unix programs that can be used to transfer files by FTP if you or your administrator will install them. I will give you some examples of ways to transfer a file by FTP while still catching all error conditions with little amounts of code.
FTPUSER is your ftp user login name
FTPPASS is your ftp password
FILE is the local file you want to upload without any path info (eg file1.txt, not /whatever/file1.txt or whatever/file1.txt
FTPHOST is the remote machine you want to FTP to
REMOTEDIR is an ABSOLUTE PATH to the location on the remote machine you want to upload to
Here are the examples:
curl --user $FTPUSER:$FTPPASS -T $FILE ftp://$FTPHOST/%2f$REMOTEDIR
ftp-upload --host $FTPHOST --user $FTPUSER --password $FTPPASS --as $REMOTEDIR/$FILE $FILE
tnftp -u ftp://$FTPUSER:$FTPPASS#$FTPHOST/%2f$REMOTEDIR/$FILE $FILE
wput $FILE ftp://$FTPUSER:$FTPPASS#$FTPHOST/%2f$REMOTEDIR/$FILE
All of these programs will return a nonzero exit code if anything at all goes wrong, along with text that indicates what failed. You can test for this and then do whatever you want with the output, log it, email it, etc as you wished.
Please note the following however:
"%2f" is used in URLs to indicate that the following path is an absolute path on the remote machine. However, if your FTP server chroots you, you won't be able to bypass this.
for the commands above that use an actual URL (ftp://etc) to the server with the user and password embedded in it, the username and password MUST be URL-encoded if it contains special characters.
In some cases you can be flexible with the remote directory being absolute and local file being just the plain filename once you are familiar with the syntax of each program. You might just have to add a local directory environment variable or just hardcode everything.
IF you really, absolutely MUST use regular FTP client, one way you can test for failure is by, inside your script, including first a command that PUTs the file, followed by another that does a GET of the same file returning it under a different name. After FTP exits, simply test for the existence of the downloaded file in your shell script, or even checksum it against the original to make sure it transferred correctly. Yeah that stinks, but in my opinion it is better to have code that is easy to read than do tons of parsing for every possible error condition. BSD FTP is just not all that great.

Here is what I finally went with. Thanks for all the help. All the answers help lead me in the right direction.
It may be a little overkill, checking both the result and the log, but it should cover all of the bases.
echo "open ftp_ip
pwd
binary
lcd /out
cd /in
mput datafile.csv
quit"|ftp -iv > ftpreturn.log
ftpresult=$?
bytesindatafile=`wc -c datafile.csv | cut -d " " -f 1`
bytestransferred=`grep -e '^[0-9]* bytes sent' ftpreturn.log | cut -d " " -f 1`
ftptransfercomplete=`grep -e '226 ' ftpreturn.log | cut -d " " -f 1`
echo "-- FTP result code: $ftpresult" >> ftpreturn.log
echo "-- bytes in datafile: $bytesindatafile bytes" >> ftpreturn.log
echo "-- bytes transferred: $bytestransferred bytes sent" >> ftpreturn.log
if [ "$ftpresult" != "0" ] || [ "$bytestransferred" != "$bytesindatafile" ] || ["$ftptransfercomplete" != "226" ]
then
echo "-- *abend* FTP Error occurred" >> ftpreturn.log
mailx -s 'FTP error' `cat email.lst` < ftpreturn.log
else
echo "-- file sent via ftp successfully" >> ftpreturn.log
fi

Why not just store all output from the command to a log file, then check the return code from the command and, if it's not 0, send the log file in the email?

Related

bash: output/write/append a csv file with timestamp and public IP

I have an R script that gets the public IP by
system("curl ifconfig.me",intern = T )
and then
writes/appends it in a CSV file
write.table(data.frame(start.script=start.time, runtime=round(Sys.time()-start.time,4), ip=myip), append = T, file = "/home/eic/ip.report.csv", row.names = F,sep = ",", col.names = !file.exists("/home/eic/ip.report.csv"))
the script runs with cron every minute.
However, i will be running it in an small raspberry Zero and the installation of R is almost 500MB
is it possible to do this with bash?
The output should create or append a CSV file with (time and public IP as strings). If the internet is not reachable , "Internet not reachable" should be output. It doesn't necessarily have to do curl ifconfig.me to check for internet connectivity . Checking for ping at 8.8.8.8 would be also an option. However it should output the public IP.
Thanks
msg=$(curl -s --max-time 3 icanhazip.com) ||
msg='Internet unreachable'
echo "$(date '+%Y-%m-%d %T %Z'),${msg:-Unkown}" >> /home/eic/ip.report.csv
Each line will look like:
2022-02-21 14:59:59,12.123.123.12 UTC
Obviously, "Internet unreachable" means "icanhazip.com unreachable". Failing to ifconfig.me, and/or ping -c 1 -W 3 google.com to log connectivity, but not IP, may be worthwhile to reduce maintenance of an embedded device.
I might even use a 5 second time out (instead of 3), for very slow connections, like bad satellite, proxies, etc.
${msg:-Unkown} replaces an empty response with Unkown.
You can change the date format: man date.
Add 2>/dev/null to curl if you don't want cron to log errors it may produce (eg if internet is down).
More info on checking internet connectivity from the shell: https://unix.stackexchange.com/questions/190513/shell-scripting-proper-way-to-check-for-internet-connectivity
#!/bin/bash
ip=$(curl --max-time 2 ifconfig.me 2>/dev/null) #Curl outputs some data on stderr. 2>/dev/null will remove that
hasInternet=$? #will be 0 if there was no error, make sure this line is directly after the curl line
curdate=$(date)
csvfile="file.csv" #right now this is a relative path to your working directory. For cron maybe better to use a absolute path
if [ $hasInternet -eq 0 ]; then
echo "${curdate},${ip}" >> $csvfile #>> will add a new line to the file
else
echo "${curdate},No internet" >> $csvfile
fi
I think this is a good start for your script. Might not be exactly as your original was, but I think you should be able to make the necessary changes.

UNIX - testing for file across ssh and returning True on original host

Need to check for distribution of a file in an array programmatically. Logging into a master server and then would like to check for file on workers using simple ssh. So far I have:
ssh $HOSTNAME "[ -e '$HOSTNAME:/directory/filename' ] && echo 'Exists'"
Based on some of the logging output, I know the ssh is successful, but how can I get the test to return a message to the master server? Running the above returns nothing.
SSH will exit with the same exit code as the command that you run on the remote host. If that command is a test, then the exit code will match what you would normally expect from a test.
I would suggest the following:
Simplify your command to only run the test over SSH
Run the echo on your local machine
It doesn't seem correct that you have $HOSTNAME: in front of your path.
ssh "$HOSTNAME" "test -e '/directory/filename'" && echo 'Exists'
I personally find if statements to be much more easily understandable, which is an optional change if you are willing to go that route:
if ssh "$HOSTNAME" "test -e '/directory/filename'"; then
echo "Exists"
else
echo "Does not exist" >&2
exit 1
fi

shell script for checking files in a directory with count

Iam trying to write a shell script to check the files in a particular path ,
if files available then i need to get success mail else I need to get failure mail .
but I my query even if 1 file is available I am getting success mail but daily I am getting 9 files , even if 1 file is not available I need to get failure mail please help me to write a script for the above logic
cd /file path
if [ -f $(date '+%Y%m%d') file name ]; then
echo "Hi Team, Input Files have been received successfully" |  mailx -s "SUCCESS" -r "FILE_CHK" userid#doamin.com
else
echo "Hi Team, Input Files have NOT been received . Please check" |  mailx -s "FAILED" -r "FILE_CHK" userid#doamin.com
fi
exit
You need to check that all the files exist and only if this is the case send the mail that it was successful. If you have only one file not present, then you should directly send the error message.
The following code prototype does the trick:
#!/bin/bash
files=( "file1" "file2" "file3" )
for i in "${files[#]}"
do
echo "Checking if file: $i exists."
if [ ! -f $i ]; then
echo "Hi Team, Input File $i has NOT been received! Please check" | mailx -s "FAILED" -r "FILE_CHK" userid#doamin.com
exit 0;
fi
done
echo "Hi Team, Input Files have been received successfully" | mailx -s "SUCCESS" -r "FILE_CHK" userid#doamin.com
Basically you have a list of files you need to check, you check element by element that it does exist and if one of the element of the list is not present then you send the failure notification by mail and you exit.
If and only if all the files are present then you send the success message!!!
Last but not least, this script is just a skeleton that you need to adapt to your particular needs (adding a timestamp etc).

Checking for a file of current date in remote server through unix shell script

I've written a script that checks for a specific file of format("OLO2OLO_$DATE.txt.zip") in the ftp server and then copies it to my local machine:
/usr/bin/ftp -n 93.179.136.9 << !EOF!
user $USR $PASSWD
cd "/0009/Codici Migrazione"
get $FILE
bye
!EOF!
echo "$FILE"
But I'm not getting the desired result from this.
This line triggers the error.
SOURCE_FOLDER="/0009/"Codici Migrazione""
It tries to execute the command Migrazione with the environment variable SOURCE_FOLDER set to /0009/Codici which doesn't exists.
What you probably wanted to do was:
SOURCE_FOLDER="/0009/Codici Migrazione"

Get the latest file from a remote server from an FTP in Unix

I need to get a file from a remote host in Unix. I am using the ftp command. The issue is I need the latest file from that location. This is how I am doing it:
dir=/home/user/nyfolders
latest_file=$(ls *abc.123.* | tail -1)
ftp -nv <<EOF
open $hostname
user $username $password
binary
cd $dir
get $latest_file
bye
EOF
But I get this error:
(remote-file) usage: get remote-file [ local-file ]
I think the way I am trying to get the file from within the ftp command is incorrect, can someone please help me out?
You cannot use shell features, like aliases, piping, variables, etc, in ftp command script.
The ftp does not support such advanced features using any syntax.
Though, you can do it two-step (to make use of shell features between the steps).
First get a listing of remote directory to a local file (/tmp/listing.txt):
ftp -nv <<EOF
open $hostname
user $username $password
cd $dir
nlist *abc.123.* /tmp/listing.txt
bye
EOF
Find the latest file:
latest_file=`tail -1 /tmp/listing.txt`
And download it:
ftp -nv <<EOF
open $hostname
user $username $password
binary
cd $dir
get $latest_file
bye
EOF

Resources