Not a long question, what does this mean?
LogMsg "File:${#}"
LogMsg() is a method that logs a message with a timestamp.
But what the heck does
${#}
mean? I should also mention the script also has $1 and $2 as well. Google produces no results.
Literally:
f() { printf '%s\n' "File: $#"; }
f "First Argument" "Second Argument" "Third Argument"
will expand to and run the command:
printf '%s\n' "File: First Argument" "Second Argument" "Third Argument"
That is to say: It expands your argument list ($1, $2, $3, etc) while maintaining separation between subsequent arguments (not throwing away any information provided by the user by way of quoting).
This is different from:
printf '%s\n' File: $#
or
printf '%s\n' File: $*
which are both the same as:
printf '%s\n' "File:" "First" "Argument" "Second" "Argument" "Third" "Argument"
...these both string-split and glob-expand the argument list, so if the user had passed, say, "*" (inside quotes intended to make it literal), the unquoted use here would replace that character with the results of expanding it as a glob, ie. the list of files in the current directory. Also, string-splitting has other side effects such as changing newlines or tabs to spaces.
It is also different from:
printf '%s\n' "File: $*"
which is the same as:
printf '%s\n' "File: First Argument Second Argument Third Argument"
...which, as you can see above, combines all arguments by putting the first character in IFS (which is by default a space) between them.
in KSH there is two positional paremeters * and #
"$*" is a single string that consists of all of the positional parameters, separated by the first character in the variable IFS (internal field separator), which is a space, TAB, and newline by default.
On the other hand, "$#" is equal to "$1" "$2" … "$N ", where N is the number of positional parameters.
For more detailed information and example : http://oreilly.com/catalog/korn2/chapter/ch04.html
This is the set of the arguments of the command line.
If you launch a script via a command like cmd a b c d, there is 5 arguments, $0 will be the command cmd, $1the first argument a, $2 the second b, etc. ${#} will be all the arguments except the command.
The one piece that was not explained by the other posts is the use of {. $# is the same as ${#} but allows you to add letters, etc if needed and those letters will not have a space added in. e.g. you could say ${foo}dog and if $foo was set to little the result would be littledog with no spaces. In the case of ${#}dogdog and $# is set to a b c d the result is "a" "b" "c" "ddogdog".
Related
I have many part-00001, part-00002, ... files.
I want to use this way:
for ((i=0;i<1000;i++)); do <some command> <formatted string with i>; done.
How can I format "part-000xx"-like string with number i in zsh?
It could be done with:
typeset -Z 5 i (using builtin typeset -Z [N])
printf "part-%05d" $i (using builtin printf "%05d" $i)
${(l:5::0:)i} (using parameter expansion flags l:expr::string1:string2:)
like below:
typeset -Z 5 j
for ((i=0;i<1000;i++)); do
# <some command> <formatted string with i>
j=$i; echo "part-$j" # use $j here for sure the effects of below 2 lines
echo "$(printf "part-%05d" $i)"
echo "part-${(l:5::0:)j}"
done
# This outputs below:
# >> part-00000
# >> part-00000
# >> part-00000
# >> part-00001
# >> part-00001
# >> part-00001
# >> ...
# >> part-00999
Here is the description for each item.
typeset
-Z [N]
Specially handled if set along with the -L flag. Otherwise, similar to -R, except that leading zeros are used for padding instead of blanks if the first non-blank character is a digit. Numeric parameters are specially handled: they are always eligible for padding with zeroes, and the zeroes are inserted at an appropriate place in the output.
-- zshbuiltins(1), typeset, Shell Builtin Commands
printf
Print the arguments according to the format specification. Formatting rules are the same as used in C.
-- zshubuiltins(1), printf, Shell Builtin Commands
l:expr::string1::string2:
Pad the resulting words on the left. Each word will be truncated if required and placed in a field expr characters wide.
The arguments :string1: and :string2: are optional; neither, the first, or both may be given. Note that the same pairs of delimiters must be used for each of the three arguments. The space to the left will be filled with string1 (concatenated as often as needed) or spaces if string1 is not given. If both string1 and string2 are given, string2 is inserted once directly to the left of each word, truncated if necessary, before string1 is used to produce any remaining padding.
-- zshexpn(1), Parameter Expansion Flags
I am trying to format a below text file, record order will be always like this
Dept 0100 Batch Load Errors for 8/16/2016 4:45:56 AM
Case 1111111111
Rectype: ABCD
Key:UMUM_REF_ID=A12345678,UMSV_SEQ_NO=1
UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
Case 2222222222
Rectype: ABCD
Key:UMUM_REF_ID=B87654321,UMSV_SEQ_NO=2
UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
NTNB ERROR :Invalid Value NTNB_MCTR_SUBJ=AMOD
Case 3333333333
Rectype: WXYZ
Key:UMUM_REF_ID=U19817250,UMSV_SEQ_NO=2
UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
as output
1111111111~ABCD~UMUM_REF_ID=A12345678,UMSV_SEQ_NO=1~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
2222222222~ABCD~UMUM_REF_ID=B87654321,UMSV_SEQ_NO=2~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID|NTNB ERROR :Invalid Value NTNB_MCTR_SUBJ=AMOD
3333333333~WXYZ~UMUM_REF_ID=U19817250,UMSV_SEQ_NO=2~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
I tried regular expression as below
sed -r '/^Case/!d;$!N;/\nRectype/!D;s/\s+$/ /;s/(.*)\n(.*)/\2\1\n\1/;P;D' file.txt
but this is working only till Rectype row, not able to achieve rest.
Thank you.
It seems to me that you're not really looking for a regular expression. You're looking for text reformatting, and you appear to have selected regular expression matching in sed as the method by which you'll process fields.
Read about XY problems here. Thankfully, you've included raw data and expected output, which is AWESOME for a new StackOverflow member. (Really! Yay you!) So I can recommend an alternative that will likely work better for you.
It is awk. Another command-line tool which, like sed, is installed on virtually every unix-like system on the planet.
$ awk -v RS= -v OFS="~" '!/^Case/{next} {sub(/^Key:/,"",$5); key=$5; for (f=6;f<=NF;f++) { if ($f=="NTNB") key=key "|"; else if ($f=="UMSV") key=key OFS; else key=key " "; key=key $f } print $2,$4,key}' inp2
1111111111~ABCD~UMUM_REF_ID=A12345678,UMSV_SEQ_NO=1~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
2222222222~ABCD~UMUM_REF_ID=B87654321,UMSV_SEQ_NO=2~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID|NTNB ERROR :Invalid Value NTNB_MCTR_SUBJ=AMOD
3333333333~WXYZ~UMUM_REF_ID=U19817250,UMSV_SEQ_NO=2~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
Here's what's going on.
awk -v RS= - This is important. It sets a "null" record separator, which tells awk that we're dealing with multi-line records. Records are terminated by a blank line, and fields within this record are separated by whitespace. (Space, tab, newline.)
-v OFS="~" - Set an output field separator of tilde, for convenience.
$1!="Case"{next} - If the current record doesn't have the word "Case" as its first field, it's not a line we can handle, so skip it.
sub(/^Key:/,"",$5); key=$5; - Trim the word Key from the beginning of the fifth field, save the field to a variable.
for (f=6;f<=NF;f++) { - Step through the remaining fields
if ($f=="NTNB") key=key "|"; - setting the appropriate field separator.
else if ($f=="UMSV") key=key OFS; - ...
else key=key " "; - Or space if the text doesn't look like a new field.
key=key $f } - Finally, add the current field to our our running variable,
print $2,$4,key} - and print everything.
NOTE: One thing this doesn't do is maintain spacing as you've shown in your "expected output" in your question. Two or more spaces will always be shrunk to just one space, since within each record, fields are separated by whitespace.
UPDATE per comments
Windows uses \r\n (CRLF) to end lines, whereas unix/linux use just \n (LF). Since your file is being generated in Windows, the "blank" lines actually contain an invisible CR, and awk never sees a record separator.
To see the "real" contents of your file, you can use tools like hexdump or od. For example:
$ printf 'foo\r\nbar\r\n' | od -c
0000000 f o o \r \n b a r \r \n
0000012
In your case, simply run:
$ od -c filename | less
(Or use more if less isn't available.)
Many systems have a package available called dos2unix which can convert text format.
If you don't have dos2unix available, you should be able to achieve the same thing using other tools. In GNU sed:
sed -i 's/\r$//' filename
Or in other sed variants, but with a shell (like bash) that supports format substitution (read man sed to see if you have a -i option):
sed $'s/\r$//' inputfile > outputfile
Or a little less precisely, as it will remove all CRs even if they're not at the end of the line, you could use tr:
tr -d '\015' < inputfile > outputfile
Or if perl is available, you can use a substitution expression that's almost identical to the one for sed (perl documentation is readily available to tell you what the options do):
perl -i -pe 's/\r\n$/\n/g' filename
Good luck!
I am attempting to use the sqrt function from awk command in my script, but all it returns is 0. Is there anything wrong with my script below?
echo "enter number"
read root
awk 'BEGIN{ print sqrt($root) }'
This is my first time using the awk command, are there any mistakes that I am not understanding here?
Maybe you can try this.
echo "enter number"
read root
echo "$root" | awk '{print sqrt($0)}'
You have to give a data input to awk. So, you can pipe 'echo'.
The BEGIN statement is to do things, like print a header...etc before
awk starts reading the input.
$ echo "enter number"
enter number
$ read root
3
$ awk -v root="$root" 'BEGIN{ print sqrt(root) }'
1.73205
See the comp.unix.shell FAQ for the 2 correct ways to pass the value of a shell variable to an awk script.
UPDATE : My proposed solution turns out to be potentially dangerous. See Ed Morton's answer for a better solution. I'll leave this answer here as a warning.
Because of the single quotes, $root is interpreted by awk, not by the shell. awk treats root as an uninitialized variable, whose value is the empty string, treated as 0 in a numeric context. $root is the root'th field of the current line -- in this case, as $0, which is the entire line. Since it's in a BEGIN block, there is no current line, so $root is the empty string -- which again is treated as 0 when passed to sqrt().
You can see this by changing your command line a bit:
$ awk 'BEGIN { print sqrt("") }'
0
$ echo 2 | awk '{ print sqrt($root) }'
1.41421
NOTE: The above is merely to show what's wrong with the original command, and how it's interpreted by the shell and by awk.
One solution is to use double quotes rather than single quotes. The shell expands variable references within double quotes:
$ echo "enter number"
enter number
$ read x
2
$ awk "BEGIN { print sqrt($x) }" # DANGEROUS
1.41421
You'll need to be careful when doing this kind of thing. The interaction between quoting and variable expansion in the shell vs. awk can be complicated.
UPDATE: In fact, you need to be extremely careful. As Ed Morton points out in a comment, this method can result in arbitrary code execution given a malicious value for $x, which is always a risk for a value read from user input. His answer avoids that problem.
(Note that I've changed the name of your shell variable from $root to $x, since it's the number whose square root you want, not the root itself.)
I tried this but it does not seem to work.
Please help thanks
TEST_STRING= test
echo Starting awk command
awk -v testString=$TEST_STRING'
BEGIN {
}
{
print testString
}
END {}
' file2
There are two problems here: You aren't actually assigning to TEST_STRING, and you're passing the program code in the same argument as the variable value. Both of these are caused by whitespace and quoting being in the wrong places.
TEST_STRING= test
...does not assign a value to TEST_STRING. Instead, it runs the command test, with an environment variable named TEST_STRING set to an empty value.
Perhaps instead you want
TEST_STRING=test
or
TEST_STRING=' test'
...if the whitespace is intentional.
Second, passing a variable to awk with -v, the right-hand side should be double-quoted, and there must be unquoted whitespace between that value and the program to be passed to awk (or other values). That is to say:
awk -v testString=$TEST_STRING' BEGIN
...will, if TEST_STRING contains no whitespace, pass the BEGIN as part of the value of testString, not as a separate argument!
awk -v testString="$TEST_STRING" 'BEGIN
...on, the other hand, ensures that the value of TEST_STRING is passed as part of the same argument as testString=, even if it contains whitespace -- and ensures that the BEGIN is passed as part of a separate argument.
I found the following bash script in order to monitor cp progress.
#!/bin/sh
cp_p()
{
strace -q -ewrite cp -- "${1}" "${2}" 2>&1 \
| awk '{
count += $NF
if (count % 10 == 0) {
percent = count / total_size * 100
printf "%3d%% [", percent
for (i=0;i<=percent;i++)
printf "="
printf ">"
for (i=percent;i<100;i++)
printf " "
printf "]\r"
}
}
END { print "" }' total_size=$(stat -c '%s' "${1}") count=0
}
I don't understand the "-ewrite" option for the strace command. The closest thing I've found is the man page for strace which is
-e write=set Perform a full hexadecimal and ASCII dump of all the
data written to file descriptors
listed in the specified set. For
example, to see all output activity on
file descriptors 3 and 5 use -e
write=3,5. Note that this is
independent from the normal tracing of
the write(2) system call which is
controlled by the option -e
trace=write.
However I don't understand what the -ewrite option does.
-ewrite means that only the "write" system call will be traced.
-e expr A qualifying expression which modifies which events
to trace or how to trace them. The format of the
expression is:
[qualifier=][!]value1[,value2]...
where qualifier is one of trace, abbrev, verbose,
raw, signal, read, or write and value is a quali-
fier-dependent symbol or number. The default qual-
ifier is trace. Using an exclamation mark negates
the set of values. For example, -eopen means lit-
erally -e trace=open which in turn means trace only
the open system call. By contrast, -etrace=!open
means to trace every system call except open. In
addition, the special values all and none have the
obvious meanings.
Note that some shells use the exclamation point for
history expansion even inside quoted arguments. If
so, you must escape the exclamation point with a
backslash.