We have requirement where i need to replace part of param value in our configuration file.
Example
key1=123-456
I need to replace the value after hyphen with new value.
I got command which is being used in other projects but i am not sure how it works.
Command
[test]$ cat test_sed_key_value.txt
key1=123-456
[test]$ sed -i -e '/key1/ s/-.*$/-789/' test_sed_key_value.txt
[test]$
[test]$ cat test_sed_key_value.txt
key1=123-789
[test]$
It will be helpful if some one can explain how the above command or is there a simpler way to do this using sed.
Here is a list of parts of that commandline, each followed by a short explanation:
sed
which tool to use
-i
flag: apply the effect directly to the processed file (whithout creating a copy of the input file)
-e
expression parameter: the sed code to apply follows
/key1/
"address": only process lines on which this regex applies, i.e. those containing the text "key1"
s/replacethis/withthis/
command: do a search-and-replace, "replacethis" and "withthis" are the next to explanations
-.*$
regex: (what is actually in the commandline instead of "replacethis") a regular expression representing a "minus" followed by anything, in any number, until the end of the line
-789
literal: (what is actually in the commandline instead of "withthis") simply that string "-789"
test_sed_key_value.txt
file parameter: process this file
I cannot think of any way to do this simpler. The shown command already uses some assumptions on the formatting of the input file.
I'd add to Yunnosch's answer that here the "replacethis" is a regexp:
-.*$
See here for an overview of the syntax of sed's regular expressions by Gnu.
Asterisk means a repetition of the previous thing, dot means any character, so .* means a sequence of characters.
$ is the end of the line.
You might want to be a bit more restrictive, since here you'd lose something in a line like this one for instance:
key1=123-456, key2=abc-def
replacing it by:
key1=123-789
removing completely the key2 part (since the .* takes all characters after the first dash until end of line).
So depending on the format of your values, you might prefer something like
-[0-9]*
(without the $), meaning a sequence of numbers after the -
or
-[0-9a-zA-Z_]
meaning a sequence of numbers or letters or underscore after the -
Related
I'm searching through text files using grep and sed commands and I also want the file names displayed before my results. However, I'm trying to remove part of the file name when it is displayed.
The file names are formatted like this: aja_EPL_1999_03_01.txt
I want to have only the date without the beginning letters and without the .txt extension.
I've been searching for an answer and it seems like it's possible to do that with a sed or a grep command by using something like this to look forward and back and extract between _ and .txt:
(?<=_)\d+(?=\.)
But I must be doing something wrong, because it hasn't worked for me and I possibly have to add something as well, so that it doesn't extract only the first number, but the whole date. Thanks in advance.
Edit: Adding also the working command I've used just in case. I imagine whatever command is needed would have to go at the beginning?
sed '/^$/d' *.txt | grep -P '(^([A-ZÖÄÜÕŠŽ].*)?[Pp][Aa][Ll]{2}.*[^\.]$)' *.txt --colour -A 1
The results look like this:
aja_EPL_1999_03_02.txt:PALLILENNUD : korraga üritavad ümbermaailmalendu kaks meeskonda
A desired output would be this:
1999_03_02:PALLILENNUD : korraga üritavad ümbermaailmalendu kaks meeskonda
First off, you might want to think about your regular expression. While the one you have you say works, I wonder if it could be simplified. You told us:
(^([A-ZÖÄÜÕŠŽ].*)?[Pp][Aa][Ll]{2}.*[^\.]$)
It looks to me as if this is intended to match lines that start with a case insensitive "PALL", possibly preceded by any number of other characters that start with a capital letter, and that lines must not end in a backslash or a dot. So valid lines might be any of:
PALLILENNUD : korraga üritavad etc etc
Õlu on kena. Do I have appalling speling?
Peeter Pall is a limnologist at EMU!
If you'd care to narrow down this description a little and perhaps provide some examples of lines that should be matched or skipped, we may be able to do better. For instance, your outer parentheses are probably unnecessary.
Now, let's clarify what your pipe isn't doing.
sed '/^$/d' *.txt
This reads all your .txt files as an input stream, deletes any empty lines, and prints the output to stdout.
grep -P 'regex' *.txt --otheroptions
This reads all your .txt files, and prints any lines that match regex. It does not read stdin.
So .. in the command line you're using right now, your sed command is utterly ignored, as sed's output is not being read by grep. You COULD instruct grep to read from both files and stdin:
$ echo "hello" > x.txt
$ echo "world" | grep "o" x.txt -
x.txt:hello
(standard input):world
But that's not what you're doing.
By default, when grep reads from multiple files, it will precede each match with the name of the file from whence that match originated. That's also what you're seeing in my example above -- two inputs, one x.txt and the other - a.k.a. stdin, separated by a colon from the match they supplied.
While grep does include the most minuscule capability for filtering (with -o, or GNU grep's \K with optional Perl compatible RE), it does NOT provide you with any options for formatting the filename. Since you can'd do anything with the output of grep, you're limited to either parsing the output you've got, or using some other tool.
Parsing is easy, if your filenames are predictably structured as they seem to be from the two examples you've provided.
For this, we can ignore that these lines contain a file and data. For the purpose of the filter, they are a stream which follows a pattern. It looks like you want to strip off all characters from the beginning of each line up to and not including the first digit. You can do this by piping through sed:
sed 's/^[^0-9]*//'
Or you can achieve the same effect by using grep's minimal filtering to return every match starting from the first digit:
grep -o '[0-9].*'
If this kind of pipe-fitting is not to your liking, you may want to replace your entire grep with something in awk that combines functionality:
$ awk '
/[\.]$/ {next} # skip lines ending in backslash or dot
/^([A-ZÖÄÜÕŠŽ].*)?PALL/ { # lines to match
f=FILENAME
sub(/^[^0-9]*/,"",f) # strip unwanted part of filename, like sed
printf "%s:%s\n", f, $0
getline # simulate the "-A 1" from grep
printf "%s:%s\n", f, $0
}' *.txt
Note that I haven't tested this, because I don't have your data to work with.
Also, awk doesn't include any of the fancy terminal-dependent colourization that GNU grep provides through the --colour option.
I have come across unix sed command usage and not able to understand what it does. Could you please help me to understand the usage ? If possible please share some reference to understand such usages of sed command.
sed -i '/^export JAVA_HOME/ s:.*:export JAVA_HOME=/usr/java/default\nexport HADOOP_PREFIX=/usr/local/hadoop\nexport HADOOP_HOME=/usr/local/hadoop\n:' $HADOOP_PREFIX/etc/hadoop/hadoop-env.sh
The command is simple, though it assumes GNU sed because of the way it uses the -i option; for macOS Sierra and related systems, you'd need to use -i '' in place of just -i.
Overall, it corresponds to:
sed -i '/Pattern/ s:.*:Replacement:' file
where:
-i means overwrite each input file with its edited output without creating a backup copy.
/Pattern/ is ^export JAVA_HOME; a line starting with the word export and then JAVA_HOME separated by a single space.
s:.*:Replacement: is a substitute command, using : instead of the more conventional / (often s/.*/Replacement/) as the pattern delimiter. This is done because the replacement text contains slashes. The .* matches the whole line. The rest of the material is written in place of the original export JAVA_HOME line. The \n sequence expands to a newline, so it actually produces a number of lines in the output.
file is $HADOOP_PREFIX/etc/hadoop/hadoop-env.sh
As others have pointed out, this is a sed command invocation. The command is short for "Stream EDitor" and is quite useful for modifying files programaticallly. Your best bet is to read the man pages (man sed, but I've broken down your particular command here for instructive purposes:
sed # The command
-i # Edit file in place (no backup)
'/^export JAVA_HOME/ # For every line that begins with 'export JAVA_HOME'...
s: # substitue...
.*: # the entire line with...
export JAVA_HOME=/usr/java/default
export HADOOP_PREFIX=/usr/local/hadoop
export HADOOP_HOME=/usr/local/hadoop
:' # End of command
$HADOOP_PREFIX/etc/hadoop/hadoop-env.sh # Run on the following file
Points of interest:
Commands can be limited to a particular address range or scope. Here, the scope was a search.
The substitue command can be delimited by almost any character (usually it is /, but in this case, : was chosen to prevent escaping of the / in the filepaths
The sed expression was enclosed in ' to prevent shell expansion of variables. Although no expansions would have taken place in this scenario, it is fairly common to see the expression wrapped in ' to eliminate the possibility.
I am very new to Unix.
I have a parameter file Parameter.prm containing following lines.
$$ErrorTable1=ErrorTable1
$$Filename1_New=FileNew.txt
$$Filename1_Old=FileOld.txt
$$ErrorTable2=ErrorTable2
$$Filename2_New=FileNew.txt
$$Filename2_Old=FileOld.txt
$$ErrorTable3=ErrorTable3
$$Filename3_New=FileNew.txt
$$Filename3_Old=FileOld.txt
I want get the output as
$$ErrorTable1=ErrorTable1
$$ErrorTable2=ErrorTable2
$$ErrorTable3=ErrorTable3
Basically, I need to delete line starting with $$Filename.
Since $ is a keyword, I am not able to interpret it as a string. How can I accomplish this using sed?
With sed:
$ sed '/$$Filename/d' infile
$$ErrorTable1=ErrorTable1
$$ErrorTable2=ErrorTable2
$$ErrorTable3=ErrorTable3
The /$$Filename/ part is the address, i.e., for all lines matching this, the command following it will be executed. The command is d, which deletes the line. Lines that don't match are just printed as is.
Extracting information from a textfile based on pattern search is a job for grep:
grep ErrorTable file
or even
grep -F '$$ErrorTable' file
-F tells grep to treat the search term as a fixed string instead of a regular expression.
Just to answer your question, if a regular expression needs to search for characters which have a special meaning in the regex language, you need to escape them:
grep '\$\$ErrorTable' file
So I've been given an assignment and the question is:
What command would you enter to see 5-letter words that begin with 'd' (upper or lower-case), followed by a lower-case vowel, and ending in 's'?
grep '^[Dd][aeiouy]..[s]' /usr/share/dict/words
^[Dd] Means that the first letter is D or d. Perfect.
[aeiouy] Means that the next letter will be one of those. Perfect.
Two dots means that the next two characters can be anything that they want. Perfect.
And s because it ends in an s. Perfect.
But when I hit enter, I'm getting things like debasements and debases. Not only are my parameters for grep being ignored, but it is reaching for too many words already, and I can't figure out what I've done wrong.
You need to anchor the end. Like this:
grep '^[Dd][aeiouy]..[s]$' /usr/share/dict/words
Otherwise you're matching all words that start with '[Dd][aeiouy]..s' which is why you get things like "dumpster"
I believe ^ and $ are string terminators, so unless the line contains ONLY the word you're looking for, you won't find it. It only works on the dictionary file but not in general files, if you try. You should use \b on both sides as they're word boundaries.
\b[Dd][aeiouy]..[s]\b
But, grep will not return you only these words. It will return you the whole line that matches the expression, for example:
~$ grep "\b[Dd][aeiouy]..[s]\b" test
aacd danis daniel danis Dunns daniedanilsanielfk
In this case, just use the parameter -o, to print only matching words, one each line.
~$ grep -o "\b[Dd][aeiouy]..[s]\b" test
danis
danis
Dunns
Basically I want to know what this line of scripting code does
function make_expand_query_string_shell {
cat <<DONE | tr '#' '#'
#!/usr/bin/ksh
DONE
cat <<DONE | tr '#' '#'
#!/usr/bin/ksh
DONE
This is a Unix "pipeline" tying together a few useful utilities to create some output.
The shell itself is going to read the first line, and break it down approximately as so:
cat — is the name of a program, which will be found on the PATH. The cat program is used to concatenate files together.
<< — is used to "redirect" the standard input to the program coming before it. Since there is nothing between cat and <<, the program will be started without any command-line parameters (e.g. filenames), and like many shell utilities, will expect its input from the "standard input" stream.
DONE is a symbol that is, essentially, a parameter to <<.
| instructs the shell to "pipe" the standard output from the program to its left (cat) to the program to its right (tr).
tr is the name of another program. Its purpose is to translate or transpose characters.
'#' '#' are command-line parameters to tr.
The << feature is called a "here-document." Every Unix program starts its life with three standard I/O streams (except under unusual circumstances) — its standard input, output, and error output. Normally, all three are connected to your terminal.
In this case, however, << will essentially link the standard input to the sequence of lines in the script file, itself, until it reads a line that matches the ending symbol given — in this case, DONE. It's called a "here-document" because the document being fed to the input is given "here" — immediately in the script file, itself.
As #KeithThompson recommended, you could have found this in the ksh manual, by searching for "<<":
<<[-]word
The shell input is read up to a line that is the same as
word after any quoting has been removed, or to an end-of-
file. No parameter substitution, command substitution,
arithmetic substitution or file name generation is per-
formed on word. The resulting document, called a here-
document, becomes the standard input. If any character
of word is quoted, then no interpretation is placed upon
the characters of the document; otherwise, parameter
expansion, command substitution, and arithmetic substitu-
tion occur, \new-line is ignored, and \ must be used to
quote the characters \, $, �. If - is appended to <<,
then all leading tabs are stripped from word and from the
document. If # is appended to <<, then leading spaces
and tabs will be stripped off the first line of the docu-
ment and up to an equivalent indentation will be stripped
from the remaining lines and from word. A tab stop is
assumed to occur at every 8 columns for the purposes of
determining the indentation.
Likewise, the | is taking the output from cat and passing it directly to the input of tr.
So, what do these two programs do? Let's check their manuals.
NAME
cat - concatenate files and print on the standard output
SYNOPSIS
cat [OPTION]... [FILE]...
DESCRIPTION
Concatenate FILE(s), or standard input, to standard output.
OK … so, this will concatenate its standard input to its standard output. What about tr?
NAME
tr - translate or delete characters
SYNOPSIS
tr [OPTION]... SET1 [SET2]
DESCRIPTION
Translate, squeeze, and/or delete characters from standard input, writ-
ing to standard output.
…
SETs are specified as strings of characters. Most represent them-
selves.
So tr well translate a character in SET1 to the character in the same position in SET2. Looks like we have two sets with only one member each, so that's easy to see what will happen.
Since cat does not do anything to its input except copy it to its output, it's being used to effectively copy the here-document as the input to tr. In turn, tr is transposing every # on its input into a #.
This creates, as its output, a typical Unix "shebang" line, of #!/usr/bin/ksh.
The entire sequence is a much more ornate version of
echo '#!/usr/bin/ksh'