I'm trying to append data to a Debian package's description during the build process.
The required data is stored in constants declared within debian/rules file.
I've override dh_gencontrol and added #printf calls which formats the data.
The issues I'm encountering are related to whitespace-containing-strings:
printf splits the given string, matching each word to an %s instance. I'd like it to use the whole string instead.
How can I manipulate said string, replacing spaces with _ and add it to the same line?
Relevant sections from debian/rules:
TEXT_STRING = "string with data"
VERSION_NUM = "v11.4"
...
...
override_dh_gencontrol:
dh_gencontrol
#echo "Adding versions to debian/control file" # This one writes to console
#printf " %-30s %-20s %s\n" "${TEXT_STRING// /_}" "${VERSION_NUM}" "${TEXT_STRING}" >> "$(CURDIR)/debian/my-package/DEBIAN/control"
Expected output:
<Package description generated by dh_gencontrol>
string_with_data v11.4 string with data
Actual output:
v11.4 string
with data
There are 2 main issues here:
Bash substitution (i.e. "${TEXT_STRING// /_}") does not work inside the make file.
Solution is to perform string manipulation within a (on-the-fly) shell session, like so:
$(shell echo ${TEXT_STRING} | sed 's/ /_/g')
String with spaces breaks up into list of words. printf sees them as multiple variables, were each one "consumes" a single %s
This results in printf adding additional lines until printing all variables.
This is solved by removing the quote marks wrapping the string, turning this "${TEXT_STRING}" to that ${TEXT_STRING}.
The final solution would be:
#printf " %-30s %-20s %s\n" $(shell echo ${TEXT_STRING} | sed 's/ /_/g') ${VERSION_NUM} ${TEXT_STRING} >> "$(CURDIR)/debian/my-package/DEBIAN/control"
Related
I'm new to zsh scripting and I was wondering if it's possible to use the sha256sum function to encrypt every value in a list.
Here is what I have tried so far:
#!/bin/zsh
filenames=`cat filenames.txt`
output='shaNames.txt'
for name in $filenames
do
echo -n $name | sha256sum >> $output
done
What I'm trying to accomplish is to encrypt every name in the list and append it to a new text file.
Any suggestions on what am I doing wrong are appreciated.
You are assigning the output of cat filenames.txt to a multiline variable. The for loop will then only loop once over the content.
What you want to do instead is e.g.:
for name in $(cat filenames.txt)
do
echo -n "$name" | sha256sum >> "$output"
done
Note that while you can still use them, backticks are deprecated in favor of $(somecommand).
Also note that you should always put variables in double quotes, as they could contain spaces.
Your method would fail anyways if one line of your textfile would contain a space.
You could use the following instead:
while read name
do
echo -n "$name" | sha256sum >> "$output"
done < filenames.txt
To anyone who might need the same. What I was doing wrong was assigning the values in the file to a single string variable instead of a list.
To correct that one must use:
filenames=(`cat filenames.txt`)
The parenthesis indicates that a list or array is stored in the filenames variable.
I am trying to format a below text file, record order will be always like this
Dept 0100 Batch Load Errors for 8/16/2016 4:45:56 AM
Case 1111111111
Rectype: ABCD
Key:UMUM_REF_ID=A12345678,UMSV_SEQ_NO=1
UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
Case 2222222222
Rectype: ABCD
Key:UMUM_REF_ID=B87654321,UMSV_SEQ_NO=2
UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
NTNB ERROR :Invalid Value NTNB_MCTR_SUBJ=AMOD
Case 3333333333
Rectype: WXYZ
Key:UMUM_REF_ID=U19817250,UMSV_SEQ_NO=2
UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
as output
1111111111~ABCD~UMUM_REF_ID=A12345678,UMSV_SEQ_NO=1~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
2222222222~ABCD~UMUM_REF_ID=B87654321,UMSV_SEQ_NO=2~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID|NTNB ERROR :Invalid Value NTNB_MCTR_SUBJ=AMOD
3333333333~WXYZ~UMUM_REF_ID=U19817250,UMSV_SEQ_NO=2~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
I tried regular expression as below
sed -r '/^Case/!d;$!N;/\nRectype/!D;s/\s+$/ /;s/(.*)\n(.*)/\2\1\n\1/;P;D' file.txt
but this is working only till Rectype row, not able to achieve rest.
Thank you.
It seems to me that you're not really looking for a regular expression. You're looking for text reformatting, and you appear to have selected regular expression matching in sed as the method by which you'll process fields.
Read about XY problems here. Thankfully, you've included raw data and expected output, which is AWESOME for a new StackOverflow member. (Really! Yay you!) So I can recommend an alternative that will likely work better for you.
It is awk. Another command-line tool which, like sed, is installed on virtually every unix-like system on the planet.
$ awk -v RS= -v OFS="~" '!/^Case/{next} {sub(/^Key:/,"",$5); key=$5; for (f=6;f<=NF;f++) { if ($f=="NTNB") key=key "|"; else if ($f=="UMSV") key=key OFS; else key=key " "; key=key $f } print $2,$4,key}' inp2
1111111111~ABCD~UMUM_REF_ID=A12345678,UMSV_SEQ_NO=1~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
2222222222~ABCD~UMUM_REF_ID=B87654321,UMSV_SEQ_NO=2~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID|NTNB ERROR :Invalid Value NTNB_MCTR_SUBJ=AMOD
3333333333~WXYZ~UMUM_REF_ID=U19817250,UMSV_SEQ_NO=2~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
Here's what's going on.
awk -v RS= - This is important. It sets a "null" record separator, which tells awk that we're dealing with multi-line records. Records are terminated by a blank line, and fields within this record are separated by whitespace. (Space, tab, newline.)
-v OFS="~" - Set an output field separator of tilde, for convenience.
$1!="Case"{next} - If the current record doesn't have the word "Case" as its first field, it's not a line we can handle, so skip it.
sub(/^Key:/,"",$5); key=$5; - Trim the word Key from the beginning of the fifth field, save the field to a variable.
for (f=6;f<=NF;f++) { - Step through the remaining fields
if ($f=="NTNB") key=key "|"; - setting the appropriate field separator.
else if ($f=="UMSV") key=key OFS; - ...
else key=key " "; - Or space if the text doesn't look like a new field.
key=key $f } - Finally, add the current field to our our running variable,
print $2,$4,key} - and print everything.
NOTE: One thing this doesn't do is maintain spacing as you've shown in your "expected output" in your question. Two or more spaces will always be shrunk to just one space, since within each record, fields are separated by whitespace.
UPDATE per comments
Windows uses \r\n (CRLF) to end lines, whereas unix/linux use just \n (LF). Since your file is being generated in Windows, the "blank" lines actually contain an invisible CR, and awk never sees a record separator.
To see the "real" contents of your file, you can use tools like hexdump or od. For example:
$ printf 'foo\r\nbar\r\n' | od -c
0000000 f o o \r \n b a r \r \n
0000012
In your case, simply run:
$ od -c filename | less
(Or use more if less isn't available.)
Many systems have a package available called dos2unix which can convert text format.
If you don't have dos2unix available, you should be able to achieve the same thing using other tools. In GNU sed:
sed -i 's/\r$//' filename
Or in other sed variants, but with a shell (like bash) that supports format substitution (read man sed to see if you have a -i option):
sed $'s/\r$//' inputfile > outputfile
Or a little less precisely, as it will remove all CRs even if they're not at the end of the line, you could use tr:
tr -d '\015' < inputfile > outputfile
Or if perl is available, you can use a substitution expression that's almost identical to the one for sed (perl documentation is readily available to tell you what the options do):
perl -i -pe 's/\r\n$/\n/g' filename
Good luck!
I have a number in the last field of my text file and I need to add a dollar sign to each line and a comma in the hundredths place of the number. So 10000 would now be $10,000.
one of the lines looks like this
World fair:399-454-9999:832 ponce Drive, Gary, IN 87878:3/22/62:24500
need it to look like this
World fair:399-454-9999:832 ponce Drive, Gary, IN 87878:3/22/62:$24,500
You can use the ' printf format flag to get the thousands groupings.
(I can't find a good reference for it but it is in the printf man page at least.)
The SUSv2 specifies one further flag character.
'
For decimal conversion (i, d, u, f, F, g, G) the output is to be grouped with thousands' grouping characters if the locale information indicates any. Note that many versions of gcc(1) cannot parse this option and will issue a warning. SUSv2 does not include %'F.
Then you just need a fairly simple application of awk.
awk -F : -v OFS=: '{$NF="$"sprintf("%\047d", $NF)}7' file
-F : sets the field separator to : so we get just the number in the final field
-v OFS=: sets the output field separator to : so awk puts the colons back for us
\047 is the octal code for a single quote to embed it in the single-quoted string easily
7 is a truth-y value to cause awk to print the line
The Perl Cookbook offers this regex solution:
sub commify {
my $text = reverse $_[0];
$text =~ s/(\d\d\d)(?=\d)(?!\d*\.)/$1,/g;
return scalar reverse $text;
}
This can be incorporated into a specific solution:
perl -lpe 'BEGIN{sub commify {$t=reverse shift; $t=~s/(\d\d\d)(?=\d)(?!\d*\.)/$1,/g; reverse $t}} s/(\d+)$/chr(044).commify($1)/e' file
output:
World fair:399-454-9999:832 ponce Drive, Gary, IN 87878:3/22/62:$24,500
A solution using unpack:
perl -lpe 'BEGIN{sub commify {$b=reverse shift; #c=unpack("(A3)*", $b); reverse join ",", #c}} s/(\d+)$/chr(044).commify($1)/e' file
If you have the Number::Format library installed, there is a shorter solution:
perl -lpe 'BEGIN{use Number::Format "format_number"} s/(\d+)$/chr(044).format_number($1)/e' file
All of the above solutions use Perl's s/foo/bar/e substitute operator with the e flag, which eval's the bar section.
chr(044) is used to print the $ (otherwise it would be eval'd)
You can add the dollar signs and the comma separately:
sed -i "s/:\([0-9]*\)$/:\$\1/g" file.txt
sed -i "s/\([0-9]\)\([0-9]\{3\}\)$/\1,\2/g" file.txt
sed -i "s/\([0-9]\)\([0-9]\{3\}\)\([0-9]\{3\}\)$/\1,\2,\3/" zeros.txt
How to replace all dots in a file with a space but dots in numbers such as 1.23232 or 4.23232 should not be replaced.
for example
Input:
abc.hello is with cdf.why with 1.9343 and 3.3232 points. What will
Output:
abc_hello is with cdf_why with 1.9343 and 3.3232 point_ what will
$ cat file
abc.hello is with cdf.why with 1.9343 and 3.3232 points. What will
this is 1.234.
here it is ...1.234... a number
.that was a number.
$ sed -e 's/a/aA/g' -e 's/\([[:digit:]]\)\.\([[:digit:]]\)/\1aB\2/g' -e 's/\./_/g' -e 's/aB/./g' -e 's/aA/a/g' file
abc_hello is with cdf_why with 1.9343 and 3.3232 points_ What will
this is 1.234_
here it is ___1.234___ a number
_that was a number_
Try any solution you're considering with that input file as it includes some edge cases (there may be more I haven't included in that file too).
The solution is basically to temporarily convert periods within numbers to some string that cannot exist anywhere else in the file so we can then convert any other periods to underscores and then undo that first temporary conversion.
So first we create a string that can't exist in the file by converting all as to the string aA which means that the string aB cannot exist in the file. Then convert all .s within numbers to aBs, then all remaining .s to _s then unwind the temporary conversions so aBs return to .s and aAs returns to as:
sed -e 's/a/aA/g' # a -> aA encode #1
-e 's/\([[:digit:]]\)\.\([[:digit:]]\)/\1aB\2/g' # 2.4 -> 2aB4 encode #2
-e 's/\./_/g' # . -> _ convert
-e 's/aB/./g' # 2aB4 -> 2.4 decode #2
-e 's/aA/a/g' # aA -> a decode #1
file
That approach of creating a temporary string that you KNOW can't exist in the file is a common alternative to picking a control character or trying to come up with some string you THINK is highly unlikely to exist in the file when you temporarily need a string that doesn't exist in the file.
I think, that will do what you want:
sed 's/\([^0-9]\)\.\([^0-9]\)/\1_\2/g' filename
This will replace all dots that are not between two digits with an underscore (_) sign (you can exchange the underscore with a space character in the above command to get spaces in the output).
If you want to write the changes back into the file, use sed -i.
Edit:
To cover dots at the beginning resp. end of the line or directly before or after a number the expression becomes a bit more ugly:
sed -r 's/(^|[^0-9])\.([^0-9]|$)/\1_\2/g;s/(^|[^0-9])\.([0-9])/\1_\2/g;s/([0-9])\.([^0-9]|$)/\1_\2/g'
resp.:
sed 's/\(^\|[^0-9]\)\.\([^0-9]\|$\)/\1_\2/g;s/\(^\|[^0-9]\)\.\([0-9]\)/\1_\2/g;s/\([0-9]\)\.\([^0-9]\|$\)/\1_\2/g'
gawk
awk -v RS='[[:space:]]+' '!/^[[:digit:]]+\.[[:digit:]]+$/{gsub("\\.", "_")}; {printf "%s", $0RT}' file.txt
since you tagged with vi, I guess you may have vim too? it would be a very easy task for vim:
:%s/\D\zs\.\ze\D/_/g
For grep there's a fixed string option, -F (fgrep) to turn off regex interpretation of the search string.
Is there a similar facility for sed? I couldn't find anything in the man. A recommendation of another gnu/linux tool would also be fine.
I'm using sed for the find and replace functionality: sed -i "s/abc/def/g"
Do you have to use sed? If you're writing a bash script, you can do
#!/bin/bash
pattern='abc'
replace='def'
file=/path/to/file
tmpfile="${TMPDIR:-/tmp}/$( basename "$file" ).$$"
while read -r line
do
echo "${line//$pattern/$replace}"
done < "$file" > "$tmpfile" && mv "$tmpfile" "$file"
With an older Bourne shell (such as ksh88 or POSIX sh), you may not have that cool ${var/pattern/replace} structure, but you do have ${var#pattern} and ${var%pattern}, which can be used to split the string up and then reassemble it. If you need to do that, you're in for a lot more code - but it's really not too bad.
If you're not in a shell script already, you could pretty easily make the pattern, replace, and filename parameters and just call this. :)
PS: The ${TMPDIR:-/tmp} structure uses $TMPDIR if that's set in your environment, or uses /tmp if the variable isn't set. I like to stick the PID of the current process on the end of the filename in the hopes that it'll be slightly more unique. You should probably use mktemp or similar in the "real world", but this is ok for a quick example, and the mktemp binary isn't always available.
Option 1) Escape regexp characters. E.g. sed 's/\$0\.0/0/g' will replace all occurrences of $0.0 with 0.
Option 2) Use perl -p -e in conjunction with quotemeta. E.g. perl -p -e 's/\\./,/gi' will replace all occurrences of . with ,.
You can use option 2 in scripts like this:
SEARCH="C++"
REPLACE="C#"
cat $FILELIST | perl -p -e "s/\\Q$SEARCH\\E/$REPLACE/g" > $NEWLIST
If you're not opposed to Ruby or long lines, you could use this:
alias replace='ruby -e "File.write(ARGV[0], File.read(ARGV[0]).gsub(ARGV[1]) { ARGV[2] })"'
replace test3.txt abc def
This loads the whole file into memory, performs the replacements and saves it back to disk. Should probably not be used for massive files.
If you don't want to escape your string, you can reach your goal in 2 steps:
fgrep the line (getting the line number) you want to replace, and
afterwards use sed for replacing this line.
E.g.
#/bin/sh
PATTERN='foo*[)*abc' # we need it literal
LINENUMBER="$( fgrep -n "$PATTERN" "$FILE" | cut -d':' -f1 )"
NEWSTRING='my new string'
sed -i "${LINENUMBER}s/.*/$NEWSTRING/" "$FILE"
You can do this in two lines of bash code if you're OK with reading the whole file into memory. This is quite flexible -- the pattern and replacement can contain newlines to match across lines if needed. It also preserves any trailing newline or lack thereof, which a simple loop with read does not.
mapfile -d '' < file
printf '%s' "${MAPFILE//"$pat"/"$rep"}" > file
For completeness, if the file can contain null bytes (\0), we need to extend the above, and it becomes
mapfile -d '' < <(cat file; printf '\0')
last=${MAPFILE[-1]}; unset "MAPFILE[-1]"
printf '%s\0' "${MAPFILE[#]//"$pat"/"$rep"}" > file
printf '%s' "${last//"$pat"/"$rep"}" >> file
perl -i.orig -pse 'while (($i = index($_,$s)) >= 0) { substr($_,$i,length($s), $r)}'--\
-s='$_REQUEST['\'old\'']' -r='$_REQUEST['\'new\'']' sample.txt
-i.orig in-place modification with backup.
-p print lines from the input file by default
-s enable rudimentary parsing of command line arguments
-e run this script
index($_,$s) search for the $s string
substr($_,$i,length($s), $r) replace the string
while (($i = index($_,$s)) >= 0) repeat until
-- end of perl parameters
-s='$_REQUEST['\'old\'']', -r='$_REQUEST['\'new\'']' - set $s,$r
You still need to "escape" ' chars but the rest should be straight forward.
Note: this started as an answer to How to pass special character string to sed hence the $_REQUEST['old'] strings, however this question is a bit more appropriately formulated.
You should be using replace instead of sed.
From the man page:
The replace utility program changes strings in place in files or on the
standard input.
Invoke replace in one of the following ways:
shell> replace from to [from to] ... -- file_name [file_name] ...
shell> replace from to [from to] ... < file_name
from represents a string to look for and to represents its replacement.
There can be one or more pairs of strings.