Confused about stdin stdout stderr? [closed] - unix

Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
So I understand the meaning of the command such as 1>&2 or 2>&1 but
my question is why do we use such commands? Redirecting the stdout to stderr or from stderr to stdout?
Also for $[$1 $sign $2] command (used for simple calculator), is $sign a default command in Unix? What does it mean? What about the $ that is outside of the brackets?

Two questions for the price of one. It would be better asked as two questions since they are almost wholly unrelated.
I/O Redirection
Why would you use 1>&2?
A standard reason is so a shell script reports errors on stderr instead of stdout. For example:
if [ ! -f "$file" ]
then
echo "$0: no such file - $file" >&2
exit 1
fi
Why would you use 2>&1?
You want to capture all of the output from a command. For example, you might be about to run a find command that will take ages, so you might write:
find $HOME -name '*perq*' -print > log.file 2>&1 &
All the data and error messages from find are sent to the log.file which you can study later when the command is complete.
The [ (test) command
You ask about $[$1 $sign $2].
This has so many problems it is difficult to answer coherently. The question about a calculator suggests that maybe $1 is supposed to be a number (let's use 13) and $2 is too (let's use 9), and $sign might be either + or -.
$[ is not normally defined as a variable, so the shell will try to find a command $[13 on the PATH and (normally) will fail.
If you had written [ $1 $sign $2 ], then we'd be better off. There is a command [ (also known as test). It is usually a shell built-in, but originally was a separate executable (and you usually find that there is a /usr/bin/test (or /bin/test) and /usr/bin/[ (or /bin/[) executable even on modern systems. Note that like all commands, the name ([) is separated from its arguments by spaces. The [ command requires the last argument to be ]. Now the expression might be:
[ 13 + 9 ]
[ 13 - 9 ]
However, the test command would object; it doesn't support arithmetic. It does support string comparisons = and !=; it supports numeric comparisons with -eq, -ne, -lt, -le, -gt or -ge.
If you want to do arithmetic, you either use the expr command, or one of the built-in alternatives:
x=$(($1 $sign $2))
This will assign 22 or 4 (depending on the value of $sign) to the variable x.

answer for redirection :
In unix the data/text file's are internally associated with something known as file handlers (integer's value) which help the OS distinguish and identify a file during processing. everything in unix is considered to be a file , even the input and output devices.
standard input is the keyboard , the standard output is stdout ie monitor which also functions as the stderr file stream , all this is by default
hence these are associated as file's hence have numbers assigned to them ....0 ,1 and 2.
so when you use something like 2>&1 you are basically telling the OS to put the stderr(2) data into the stdout(1) data stream
answer for $ sign :
the dollar sign here is a way to access the value of the variables.
example when you have a variable a like as below ::
Nitin#Kaizen > a=5
Nitin#Kaizen > echo a --> will just print a
Nitin#Kaizen > echo $a --> will print 5
note : $a is same as ${a} or $[a] , they access the value of a variable .... kind of de-referencing
$[$1 $sign $2] :: you issue statement
Nitin#Kaizen > a=5
Nitin#Kaizen > b=3
Nitin#Kaizen > sign=+
Nitin#Kaizen > echo `expr $[ $a $sign $b ]`
8 --- output is same as 5 + 3 , the value of a + b
answer for why you are confused :
you need make a slight effort to read a book completely again. these are very basics and hence become clear with a bit of programing and study.
Any good unix basic book would do that covers the chapters on File System (UFS) and a bit on scripting will do !!
hope this helps.

Related

/usr/xpg4/bin/grep -q [^0-9] does not always work as expected

I have a Unix ksh script that has been in daily use for years (kicked off at night by the crontab). Recently one function in the script is behaving erratically as never happened before. I tried various ways to find out why, but have no success.
The function validates an input string, which is supposed to be a string of 10 numeric characters. The function checks if the string length is 10, and whether it contains any non-numeric characters:
#! /bin/ksh
# The function:
is_valid_id () {
# Takes one argument, which is the ID being tested.
if [[ $(print ${#1}) -ne 10 ]] || print "$1" | /usr/xpg4/bin/grep -q [^0-9] ; then
return 1
else
return 0
fi
}
cat $input_file | while read line ; do
id=$(print $line | awk -F: '{print $5}')
# Calling the function:
is_valid_id $id
stat=$?
if [[ $stat -eq 1 ]] ; then
print "The ID $id is invalid. Request rejected.\n" >> $ERRLOG
continue
else
...
fi
done
The problem with the function is that, every night, out of scores or hundreds of requests, it finds the IDs in several requests as invalid. I visually inspected the input data and found that all the "invalid" IDs are actually strings of 10 numeric characters as should be. This error seems to be random, because it happens with only some of the requests. However, while the rejected requests persistently come back, it is consistently the same IDs that are picked out as invalid day after day.
I did the following:
The Unix machine has been running for almost a year, therefore might need to be refreshed. The system admin to reboot the machine at my request. But the problem persists after the reboot.
I manually ran exactly the same two tests in the function, at command prompt, and the IDs that have been found invalid at night are all valid.
I know the same commands may behave differently invoked manually or in a script. To see how the function behaves in script, the above code excerpt is the small script I ran to reproduce the problem. And indeed, some (though not all) of the IDs found to be invalid at night are also found invalid by the small trouble-shooting script.
I then modified that troubleshooting script to run the two tests one at a time, and found it is the /usr/xpg4/bin/grep -q [^0-9] test that erroneously finds some of the ID as containing non-numeric character(s). Well, the IDs are all numeric characters, at least visually.
I checked if there is any problem with the xpg4 grep command file (ls -l /usr/xpg4/bin/grep), to see if it is put there recently. But its timestamp is year 2005 (this machine runs Solaris 10).
Knowing that the data comes from a central ERP system, to which data entry is performed from different locations using all kinds of various terminal machines running all kinds of possible operating systems that support various character sets and encodings. The ERP system simply allows them. But can characters from other encodings visually appear as numeric characters but the encoded values are not as the /usr/xpg4/bin/grep command expects to be on our Unix machine? I tried the od (octal dump) command but it does not help me much as I am not familiar with it. Maybe I need to know more about od for solving this problem.
My temporary work-around is omitting the /usr/xpg4/bin/grep -q [^0-9] test. But the problem has not been solved. What can I try next?
Your validity test function happens to be more complicated than it should be. E.g. why do you use a command substitution with print for ${#1}? Why don't you use ${#1} directly? Next, forking grep to test for a non-number is a slow and expensive operation. What about this equivalent function, 100% POSIX and blazingly fast:
is_valid_id () {
# Takes one argument, which is the ID being tested.
if test ${#1} -ne 10; then
return 1 # ID length not exactly 10.
fi
case $1 in
(*[!0-9]*) return 1;; # ID contains a non-digit.
(*) return 0;; # ID is exactly 10 digits.
esac
}
Or even more simple, if you don't mind repeating yourself:
is_valid_id () {
# Takes one argument, which is the ID being tested.
case $1 in
([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]) # 10 digits.
return 0;;
(*)
return 1;;
esac
}
This also avoids your unquoted use of a grep pattern, which is error-prone in the presence of one-character file names. Does this work better?

Parsing variable in loop incorrectly [duplicate]

I want to run certain actions on a group of lexicographically named files (01-09 before 10). I have to use a rather old version of FreeBSD (7.3), so I can't use yummies like echo {01..30} or seq -w 1 30.
The only working solution I found is printf "%02d " {1..30}. However, I can't figure out why can't I use $1 and $2 instead of 1 and 30. When I run my script (bash ~/myscript.sh 1 30) printf says {1..30}: invalid number
AFAIK, variables in bash are typeless, so how can't printf accept an integer argument as an integer?
Bash supports C-style for loops:
s=1
e=30
for i in ((i=s; i<e; i++)); do printf "%02d " "$i"; done
The syntax you attempted doesn't work because brace expansion happens before parameter expansion, so when the shell tries to expand {$1..$2}, it's still literally {$1..$2}, not {1..30}.
The answer given by #Kent works because eval goes back to the beginning of the parsing process. I tend to suggest avoiding making habitual use of it, as eval can introduce hard-to-recognize bugs -- if your command were whitelisted to be run by sudo and $1 were, say, '$(rm -rf /; echo 1)', the C-style-for-loop example would safely fail, and the eval example... not so much.
Granted, 95% of the scripts you write may not be accessible to folks executing privilege escalation attacks, but the remaining 5% can really ruin one's day; following good practices 100% of the time avoids being in sloppy habits.
Thus, if one really wants to pass a range of numbers to a single command, the safe thing is to collect them in an array:
a=( )
for i in ((i=s; i<e; i++)); do a+=( "$i" ); done
printf "%02d " "${a[#]}"
I guess you are looking for this trick:
#!/bin/bash
s=1
e=30
printf "%02d " $(eval echo {$s..$e})
Ok, I finally got it!
#!/bin/bash
#BSD-only iteration method
#for day in `jot $1 $2`
for ((day=$1; day<$2; day++))
do
echo $(printf %02d $day)
done
I initially wanted to use the cycle iterator as a "day" in file names, but now I see that in my exact case it's easier to iterate through normal numbers (1,2,3 etc.) and process them into lexicographical ones inside the loop. While using jot, remember that $1 is the numbers amount, and the $2 is the starting point.

Facing issue while invoking grep inside awk command

I am looking for extracting some information from log using awk and based on the information returned i want to grep the whole file and write all the output from gerp and awk to a file. I was able to extract few information form awk but while using grep inside awk i am not able to extract information. Please find the logs as follow.
2014-04-10 13:55:59,837 [WebContainer : 4] [com.cisco.ata.service.AtAService] WARN - AtAService::AtAServiceRequest DetailMessage - module=ataservice;service=ataservicerequest;APP_ID=CDCSDSATAUser.gen;VIEW_NAME=/EntitlementView[CCOID="frhocevar"]REQUEST_ID_STRING=-105411838, took 100 milliseconds.
Based on the REQUEST_ID_STRING i have to get usecaseID.
2014-04-10 13:55:59,800 [Thread-66] [com.cisco.ata.cla.CLAManager] INFO - CLAManager.getAttributeFromCLAMapping() took 6 ms, for useCaseID - UC41, condition= (CCOID=frhocevar), requestID= -105411838
i am extracting REQUEST_ID_STRING using awk but i am not able to extract "useCaseID" using grep.
Below is the command i am using.
grep -i -r 'AtAService::AtAServiceRequest DetailMessage - module=ataservice;service=ataservicerequest' /opt/httpd/logs/apps/atasvc/prod1/was70/*/*.log* |
awk 'BEGIN{count=0;}{if($14>1000){print $0}}' |
awk 'BEGIN{ FS=";"}
{a = substr($3,8)}
{b = substr($4,index($4,"/")+1,index($4,"]R")-index($4,"/"))}
{c = substr($4,index($4,"G=")+2,index($4,", took")-index($4,"G=")-2);}
{d = substr($1,0,index($1,":")-1)}
{e=grep command which will extract usecaseid from $d having file name}
{ print a","b","c","d","e} '
Please help me in this issue.
Thanks in advance
I'm incredibly tired, so this is likely not the best solution, but it uses some basic "awkisms" that make for some pretty good boilerplate starting points for a lot of stuff.
AirBoxOmega:~ d$ cat log
2014-04-10 13:55:59,837 [WebContainer : 4] [com.cisco.ata.service.AtAService] WARN - AtAService::AtAServiceRequest DetailMessage - module=ataservice;service=ataservicerequest;APP_ID=CDCSDSATAUser.gen;VIEW_NAME=/EntitlementView[CCOID="frhocevar"]REQUEST_ID_STRING=-105411838, took 100 milliseconds.
2014-04-10 13:55:59,800 [Thread-66] [com.cisco.ata.cla.CLAManager] INFO - CLAManager.getAttributeFromCLAMapping() took 6 ms, for useCaseID - UC41, condition= (CCOID=frhocevar), requestID= -105411838
AirBoxOmega:~ d$ cat stackHelp.awk
{
if ($0 ~ /AtAService::AtAServiceRequest DetailMessage/ && $(NF - 1) > 99) {
split($0, tmp, "[-,]")
slow[tmp[7]]++
}
if (slow[substr($NF,2)]) {
split($0, tmp, "[-,]")
print $NF tmp[8]
}
}
AirBoxOmega:~ d$ gawk -f stackHelp.awk log
-105411838 UC41
This uses a pretty basic awk concept where if you know that there is something common among your log lines (a sessionID, or some such) you create an array for that based on certain conditions (in this case that the log line contains a given string and that the next to last column is > 99). Then later when you run into that same sessionID, you can check to see if an array exists for it, and if so, pull out even more info.
You may need/want to add something to the second if statement so it's only checking log lines you care about, but honestly, awk is so fast that it probably won't matter. (I'm using gawk [via brew] as the version of awk that comes with OSX is somewhat lacking, but this code is basic enough that awk or gawk should work.)
If you need a better explanation of the code, I'll try to explain better.
Ninja edit: Few quit tips:
Don't use grep -i unless you really don't know the case you're looking for. Case insensitivity will make your searches MUCH slower
If you're not using any kind of regular expressions, use fgrep instead of grep. It's much faster out the box.
Learn how to ask questions efficiently. Your question was pretty clear, but use tags to make the log lines more readable, and remember that every technical question should include:
What your input is
What your output should be
What you tried
What you expected
What you got
Get good at awk. The world is slowly moving away from command line centric stuff, and people may say it's not worth it, but once you understand basic concepts in awk, it's easy to apply them elsewhere, be it python, log utilities, or just thinking in terms of data aggregation.

How can I tell if a makefile is being run from an interactive shell?

I have a makefile which runs commands that can take a while. I'd like those commands to be chatty if the build is initiated from an interactive shell but quieter if not (specifically, by cron). Something along the lines of (pseudocode):
foo_opts = -a -b -c
if (make was invoked from an interactive shell):
foo_opts += --verbose
all: bar baz
foo $(foo_opts)
This is GNU make. If the specifics of what I'm doing matter, I can edit the question.
It isn't strictly determining whether it is invoked from an interactive shell or not, but for a cron job in which the output is redirected to a file, the answer to this question would be the same as for How to detect if my shell script is running through a pipe?:
if [ -t 0 ]
then
# input is from a terminal
fi
Edit: To use this to set a variable in a Makefile (in GNU make, that is):
INTERACTIVE:=$(shell [ -t 0 ] && echo 1)
ifdef INTERACTIVE
# is a terminal
else
# cron job
endif
http://www.faqs.org/faqs/unix-faq/faq/part5/section-5.html
5.5) How can I tell if I am running an interactive shell?
In the C shell category, look for the variable $prompt.
In the Bourne shell category, you can look for the variable $PS1,
however, it is better to check the variable $-. If $- contains
an 'i', the shell is interactive. Test like so:
case $- in
*i*) # do things for interactive shell
;;
*) # do things for non-interactive shell
;;
esac
I do not think you can easily find out. I suggest adopting an alternative strategy, probably by quelling the verbose output from the cron job. I would look to do that using a makefile like this:
VERBOSE = --verbose
foo_opts = -a -b -c ${VERBOSE}
all: bar baz
foo $(foo_opts)
Then, in the cron job, specify:
make VERBOSE=
This command-line specification of VERBOSE overrides the one in the makefile (and cannot be changed by the makefile). That way, the specialized task (cron job) that you set up once and use many times will be done without the verbose output; the general task of building will be done verbosely (unless you elect to override the verbose-ness on the command line).
One minor advantage of this technique is that it will work with any variant of make; it does not depend on any GNU Make facility.
I’m not really sure what "am interactive" means. Do you mean if you have a valid /dev/tty? If so, then you could check that. Most of us check isatty on stdin, though, because it answers the questions we want to know: is there someone there to type something.
Just a note: you can also see the related discussion that I had about detecting redirection of STDOUT from inside a Makefile.
I believe it will be helpful to readers of this question - executive summary:
-include piped.mk
all: piped.mk
ifeq ($(PIPED),1)
#echo Output of make is piped because PIPED is ${PIPED}
else
#echo Output of make is NOT piped because PIPED is ${PIPED}
endif
#rm -f piped.mk
piped.mk:
#[ -t 1 ] && PIPED=0 || PIPED=1 ; echo "PIPED=$${PIPED}" > piped.mk
$ make
Output of make is NOT piped because PIPED is 0
$ make | more
Output of make is piped because PIPED is 1
In my answer there I explain why the [-t 1] has to be done in an action and not in a variable assignment (as in the recommended answer here), as well as the various pitfalls regarding re-evaluation of a generated Makefile (i.e. the piped.mk above).
The term interactive in this question seems to imply redirection of STDIN... in which case replacing [ -t 1 ] with [ -t 0 ] in my code above should work as-is.
Hope this helps.

ksh: how to probe stdin?

I want my ksh script to have different behaviors depending on whether there is something incoming through stdin or not:
(1) cat file.txt | ./script.ksh (then do "cat <&0 >./tmp.dat" and process tmp.dat)
vs. (2) ./script.ksh (then process $1 which must be a readable regular file)
Checking for stdin to see if it is a terminal[ -t 0 ] is not helpful, because my script is called from an other script.
Doing "cat <&0 >./tmp.dat" to check tmp.dat's size hangs up waiting for an EOF from stdin if stdin is "empty" (2nd case).
How to just check if stdin is "empty" or not?!
EDIT: You are running on HP-UX
Tested [ -t 0 ] on HP-UX and it appears to be working for me. I have used the following setup:
/tmp/x.ksh:
#!/bin/ksh
/tmp/y.ksh
/tmp/y.ksh:
#!/bin/ksh
test -t 0 && echo "terminal!"
Running /tmp/x.ksh prints: terminal!
Could you confirm the above on your platform, and/or provide an alternate test setup more closely reflecting your situation? Is your script ultimately spawned by cron?
EDIT 2
If desperate, and if Perl is available, define:
stdin_ready() {
TIMEOUT=$1; shift
perl -e '
my $rin = "";
vec($rin,fileno(STDIN),1) = 1;
select($rout=$rin, undef, undef, '$TIMEOUT') < 1 && exit 1;
'
}
stdin_ready 1 || 'stdin not ready in 1 second, assuming terminal'
EDIT 3
Please note that the timeout may need to be significant if your input comes from sort, ssh etc. (all these programs can spawn and establish the pipe with your script seconds or minutes before producing any data over it.) Also, using a hefty timeout may dramatically penalize your script when there is nothing on the input to begin with (e.g. terminal.)
If potentially large timeouts are a problem, and if you can influence the way in which your script is called, then you may want to force the callers to explicitly instruct your program whether stdin should be used, via a custom option or in the standard GNU or tar manner (e.g. script [options [--]] FILE ..., where FILE can be a file name, a - to denote standard input, or a combination thereof, and your script would only read from standard input if - were passed in as a parameter.)
This strategy works for bash, and would likely work for ksh. Poll 'tty':
#!/bin/bash
set -a
if [ "$( tty )" == 'not a tty' ]
then
STDIN_DATA_PRESENT=1
else
STDIN_DATA_PRESENT=0
fi
if [ ${STDIN_DATA_PRESENT} -eq 1 ]
then
echo "Input was found."
else
echo "Input was not found."
fi
Why not solve this in a more traditional way, and use the command line argument to indicate that the data will be coming from stdin?
For an example, consider the difference between:
echo foo | cat -
and
echo foo > /tmp/test.txt
cat /tmp/test.txt

Resources