output of one command is argument of another - unix
Is there any way to fit in 1 line using the pipes the following:
output of
sha1sum $(xpi) | grep -Eow '^[^ ]+'
goes instead of 456
sed 's/#version#/456/' input.txt > output.txt
Um, I think you can nest $(command arg arg) occurances, so if you really need just one line, try
sed "s/#version#/$(sha1sum $(xpi) | grep -Eow '^[^ ]+')/" input.txt \
> output.txt
But I like Trey's solution putting it one two lines; it's less confusing.
This is not possible using pipes. Command nesting works though:
sed 's/#version#/'$(sha1sum $(xpi) | grep -Eow '^[^ ]+')'/' input.txt > output.txt
Also note that if the results of the nested command contain the / character you will need to use a different character as delimiter (#, |, $, and _ are popular ones) or somehow escape the forward slashes in your string. This StackOverflow question also has a solution to the escaping problem. The problem can be solved by piping the command to sed and replacing all forward slashes (for escape characters) and backslashes (to avoid conflicts with using / as the outer sed delimiter).
The following regular expression will escape all \ characters and all / characters in the command:
sha1sum $(xpi) | grep -Eow '^[^ ]+' | sed -e 's/\(\/\|\\\|&\)/\\&/g'
Nesting this as we did above we get this solution which should properly escape slashes where needed:
sed 's/#version#/'$(sha1sum $(xpi) | grep -Eow '^[^ ]+' | sed -e 's/\(\/\|\\\|&\)/\\&/g')'/' input.txt > output.txt
Personally I think that looks like a mess as one line, but it works.
Related
Unix command to parse string
I'm trying to figure out a command to parse the following file content: Operation=GET Type=HOME Counters=CacheHit=0,Exception=1,Validated=0 I need to extract Exception=1 into its own line. I'm fiddling with awk, sed and grep but not making much progress. Does anyone have any tips on using any unix command to perform this? Thanks
Since your file is close to bash syntax, there is a fun little trick you can do to make bash itself parse the file. First, use some program like tr to transform the input into a something bash can parse, and then "source" that, which will create shell variables you can expand later to get the values. source <(tr , $'\n' < file_name_goes_here) echo $Exception
Many ways to do this. Here is one assuming the file is called "file.txt". Grab the line you want, replace everything from the start of the line up to Except with just Except, then pull out the first field using comma as the delimiter.
$ grep Exception file.txt | sed 's/.*Except/Except/g' | cut -d, -f 1
Exception=1
If you wanted to use gawk:
$ grep Exception file.txt | sed 's/.*Except/Except/g' | gawk -F, '{print $1}'
Exception=1
or just using grep and sed:
$ grep Exception file.txt | sed 's/.*\(Exception=[0-9]*\).*/\1/g'
Exception=1
or as #sheltter reminded me:
$ egrep -o "Exception=[0-9]+" file.txt
Exception=1
No need to use a mix of commands.
awk -F, 'NR==2 {print RS$1}' RS="Exception" file
Exception=1
Here we split the line by the keyword we look for RS="Exception"
If the line has two record (only when keyword is found), then
print first field, separated using command, with Record selector.
PS This only works if you have one Exception field
How to use sed to extract text between two bar signs (i.e. '|')?
I would like to extract text that falls between two | signs in a file with multiple lines. For instance, I want to extract P16 from sp|P16|SM2. I have found a possible answer here. However, I cannot apply the answer to my case. I am using the following: sed -n '/|/,/|/ p' filename or this by escaping the | sign: sed -n '/\|/,/\|/ p' filename But what I receive as result are all the lines in the file unchanged even though I am using -n to suppress automatic printing of pattern space. Any ideas what I am missing? [EDIT]: I can get the desired result using the following. However, I would like an explanation why the above mentioned is not working: sed 's/^sp|//' filename | sed 's/|.*//'
the tool for this task is cut $ echo "sp|P16|SM2" | cut -d'|' -f2 P16
awk is better choice for column based data:
awk -F'|' '{print $2}'
will give you P16
sed one-liner:
The following sed one-liner will only leave the 2nd column for you:
kent$ echo "sp|P16|SM2"|sed 's/[^|]*|//;s/|[^|]*//'
P16
Or using grouping:
kent$ echo "sp|P16|SM2"|sed 's/.*|\([^|]*\)|.*/\1/'
P16
Short explanation why your two commands didn't work:
1) sed -n '/|/,/|/ p' filename
This sed will print lines between two lines which containing |
2) sed -n '/\|/,/\|/ p' filename
Sed takes BRE as default. If you escape the |, you gave them special meaning, the logical OR. again, the /pat1/,/pat2/ address was wrong usage for your case, it checks lines, not within a line.
How to remove blank lines from a Unix file
I need to remove all the blank lines from an input file and write into an output file. Here is my data as below. 11216,33,1032747,64310,1,0,0,1.878,0,0,0,1,1,1.087,5,1,1,18-JAN-13,000603221321 11216,33,1033196,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,059762153003 11216,33,1033246,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,000603211032 11216,33,1033280,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,055111034001 11216,33,1033287,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000378689701 11216,33,1033358,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000093737301 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041926 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041954 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049326 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049383 11216,33,1036985,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000093415580 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781202001 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781261305 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781603955 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781615746
sed -i '/^$/d' foo This tells sed to delete every line matching the regex ^$ i.e. every empty line. The -i flag edits the file in-place, if your sed doesn't support that you can write the output to a temporary file and replace the original: sed '/^$/d' foo > foo.tmp mv foo.tmp foo If you also want to remove lines consisting only of whitespace (not just empty lines) then use: sed -i '/^[[:space:]]*$/d' foo Edit: also remove whitespace at the end of lines, because apparently you've decided you need that too: sed -i '/^[[:space:]]*$/d;s/[[:space:]]*$//' foo
awk 'NF' filename awk 'NF > 0' filename sed -i '/^$/d' filename awk '!/^$/' filename awk '/./' filename The NF also removes lines containing only blanks or tabs, the regex /^$/ does not.
Use grep to match any line that has nothing between the start anchor (^) and the end anchor ($): grep -v '^$' infile.txt > outfile.txt If you want to remove lines with only whitespace, you can still use grep. I am using Perl regular expressions in this example, but here are other ways: grep -P -v '^\s*$' infile.txt > outfile.txt or, without Perl regular expressions: grep -v '^[[:space:]]*$' infile.txt > outfile.txt
sed -e '/^ *$/d' input > output Deletes all lines which consist only of blanks (or is completely empty). You can change the blank to [ \t] where the \t is a representation for tab. Whether your shell or your sed will do the expansion varies, but you can probably type the tab character directly. And if you're using GNU or BSD sed, you can do the edit in-place, if that's what you want, with the -i option. If I execute the above command still I have blank lines in my output file. What could be the reason? There could be several reasons. It might be that you don't have blank lines but you have lots of spaces at the end of a line so it looks like you have blank lines when you cat the file to the screen. If that's the problem, then: sed -e 's/ *$//' -e '/^ *$/d' input > output The new regex removes repeated blanks at the end of the line; see previous discussion for blanks or tabs. Another possibility is that your data file came from Windows and has CRLF line endings. Unix sees the carriage return at the end of the line; it isn't a blank, so the line is not removed. There are multiple ways to deal with that. A reliable one is tr to delete (-d) character code octal 15, aka control-M or \r or carriage return: tr -d '\015' < input | sed -e 's/ *$//' -e '/^ *$/d' > output If neither of those works, then you need to show a hex dump or octal dump (od -c) of the first two lines of the file, so we can see what we're up against: head -n 2 input | od -c Judging from the comments that sed -i does not work for you, you are not working on Linux or Mac OS X or BSD — which platform are you working on? (AIX, Solaris, HP-UX spring to mind as relatively plausible possibilities, but there are plenty of other less plausible ones too.) You can try the POSIX named character classes such as sed -e '/^[[:space:]]*$/d'; it will probably work, but is not guaranteed. You can try it with: echo "Hello World" | sed 's/[[:space:]][[:space:]]*/ /' If it works, there'll be three spaces between the 'Hello' and the 'World'. If not, you'll probably get an error from sed. That might save you grief over getting tabs typed on the command line.
grep . file grep looks at your file line-by-line; the dot . matches anything except a newline character. The output from grep is therefore all the lines that consist of something other than a single newline.
with awk awk 'NF > 0' filename
To be thorough and remove lines even if they include spaces or tabs something like this in perl will do it: cat file.txt | perl -lane "print if /\S/" Of course there are the awk and sed equivalents. Best not to assume the lines are totally blank as ^$ would do. Cheers
You can sed's -i option to edit in-place without using temporary file: sed -i '/^$/d' file
unexpected EOF while looking for matching `"'
I am writing a script which has command to execute as below:
cat /abc | grep -v ^# | grep -i root | sed -e '\''s/"//g'\'' | awk '\''{print $2}'\''
When running the script on SunOS, i am getting below error:
test: line 1: unexpected EOF while looking for matching `"'
test: line 3: syntax error: unexpected end of file
Tried with different option.. but no luck.
Need somebody help me identify what is missing in the above command.
what are those escapes ?!
cat /abc | grep -v '^#' | grep -i root | sed -e '\''s/"//g'\'' | awk '\''{print $2}'\''
^ ^ ^ ^
Your problem is there:
sed -e '\''s/"//g'\''
^ unmatched
The quoting is all wrong. Why do you use single quote, backslash, single quote, single quote,and always in that order? Regardless, you have an unquoted double quote, so the shell expects you to add a closing quote for the quoted string which starts with that opening double quote.
As a matter of style, you should also lose the Useless Use of Cat, and think about how to simplify your script. At least:
grep -v ^# /abc | grep -i root | sed -e 's/"//g' | awk '{print $2}'
... but in practice
awk '/^#/ { next } /[Rr][Oo][Oo][Tt]/ { gsub ("\"",""); print $2 }' /abc
Because some of the characters in the awk and sed scripts have a special meaning to the shell, we put them in single quotes. If you need to have single quotes in a script, you need to double quote them; a frequent pattern is to have a string in single quotes adjacent to a string in double quotes, like this: echo '"'"'". This echos " (quoted in single quotes) immediately followed by ' (quoted in double quotes).
Edit Updated analysis of quoting problem; added code example; corrected code example. Final edit corrects quoting of gsub in awk script, and adds a small discussion of quoting.
How to extract text between two words in unix?
I am using basic sed expression :- sed -n "am/,/sed/p" to get the text between "am" and "sed" which will output "am \n using \n basic \n sed". But my real problem is if the string would be :- I am using basic grep expression. I applied the above sed in this sentence then it gave "am \n using \n basic \n grep \n expression" which it should not give it. How to discard the output if there would be no matching? Any suggestions?
The command in the question (sed -n "/am/,/sed/p", note the added slash) means: Find a line containing the string am and print (p) until a line containing sed occurs Therefore it prints: I am using basic grep expression because it contains am. If you would add some more lines they will be printed, too, until a line containing sed occurs. E.g.: echo -e 'I am using basic grep expression.\nOne more line\nOne with sed\nOne without' | sed -n "/am/,/sed/p" results in: I am using basic grep expression. One more line One with sed I think - what you want to do is something like that: sed -n "s/.*\(am.*sed\).*/\1/p" Example: echo 'I am using basic grep expression.' | sed -n "s/.*\(am.*sed\).*/\1/p" echo 'I am using basic sed expression.' | sed -n "s/.*\(am.*sed\).*/\1/p" sed -n "s/.*\(am.*sed\).*/\1/p"
You have to use slightly different sed command like:
sed -n '/am/{:a; /am/x; $!N; /sed/!{$!ba;}; /sed/{s/\n/ /gp;}}' file
To print ONLY lines that contain text am and sed spanned across multiple lines.
When Using SED this can work but it's quite an overwhelming syntax...
if you need to crop part of a multi-line (\n) text, you might want to try a simpler way using grep:
cat multi_line.txt | grep -oP '(?s)(?<=START phrase).*(?=END phrase)'
For example, I find this as the easiest way to grab perforce changelist description (without rest of CL info):
p4 describe {CL NUMBER} | grep -oP '(?s).*(?=Affected files)'
Note, you can play with the <= and >= to include or not include, the starting/ending phrases in the output.