Text file formatting using regular expression - unix

Question

I am trying to format a below text file, record order will be always like this
Dept 0100 Batch Load Errors for 8/16/2016 4:45:56 AM
Case 1111111111
Rectype: ABCD
Key:UMUM_REF_ID=A12345678,UMSV_SEQ_NO=1
UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
Case 2222222222
Rectype: ABCD
Key:UMUM_REF_ID=B87654321,UMSV_SEQ_NO=2
UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
NTNB ERROR :Invalid Value NTNB_MCTR_SUBJ=AMOD
Case 3333333333
Rectype: WXYZ
Key:UMUM_REF_ID=U19817250,UMSV_SEQ_NO=2
UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
as output
1111111111~ABCD~UMUM_REF_ID=A12345678,UMSV_SEQ_NO=1~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
2222222222~ABCD~UMUM_REF_ID=B87654321,UMSV_SEQ_NO=2~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID|NTNB ERROR :Invalid Value NTNB_MCTR_SUBJ=AMOD
3333333333~WXYZ~UMUM_REF_ID=U19817250,UMSV_SEQ_NO=2~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
I tried regular expression as below
sed -r '/^Case/!d;$!N;/\nRectype/!D;s/\s+$/ /;s/(.*)\n(.*)/\2\1\n\1/;P;D' file.txt
but this is working only till Rectype row, not able to achieve rest.
Thank you.

Answer 1

It seems to me that you're not really looking for a regular expression. You're looking for text reformatting, and you appear to have selected regular expression matching in sed as the method by which you'll process fields.
Read about XY problems here. Thankfully, you've included raw data and expected output, which is AWESOME for a new StackOverflow member. (Really! Yay you!) So I can recommend an alternative that will likely work better for you.
It is awk. Another command-line tool which, like sed, is installed on virtually every unix-like system on the planet.
$ awk -v RS= -v OFS="~" '!/^Case/{next} {sub(/^Key:/,"",$5); key=$5; for (f=6;f<=NF;f++) { if ($f=="NTNB") key=key "|"; else if ($f=="UMSV") key=key OFS; else key=key " "; key=key $f } print $2,$4,key}' inp2
1111111111~ABCD~UMUM_REF_ID=A12345678,UMSV_SEQ_NO=1~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
2222222222~ABCD~UMUM_REF_ID=B87654321,UMSV_SEQ_NO=2~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID|NTNB ERROR :Invalid Value NTNB_MCTR_SUBJ=AMOD
3333333333~WXYZ~UMUM_REF_ID=U19817250,UMSV_SEQ_NO=2~UMSV ERROR :UNITS_ALLOW must be > or = UNITS_PAID
Here's what's going on.
awk -v RS= - This is important. It sets a "null" record separator, which tells awk that we're dealing with multi-line records. Records are terminated by a blank line, and fields within this record are separated by whitespace. (Space, tab, newline.)
-v OFS="~" - Set an output field separator of tilde, for convenience.
$1!="Case"{next} - If the current record doesn't have the word "Case" as its first field, it's not a line we can handle, so skip it.
sub(/^Key:/,"",$5); key=$5; - Trim the word Key from the beginning of the fifth field, save the field to a variable.
for (f=6;f<=NF;f++) { - Step through the remaining fields
if ($f=="NTNB") key=key "|"; - setting the appropriate field separator.
else if ($f=="UMSV") key=key OFS; - ...
else key=key " "; - Or space if the text doesn't look like a new field.
key=key $f } - Finally, add the current field to our our running variable,
print $2,$4,key} - and print everything.
NOTE: One thing this doesn't do is maintain spacing as you've shown in your "expected output" in your question. Two or more spaces will always be shrunk to just one space, since within each record, fields are separated by whitespace.
UPDATE per comments
Windows uses \r\n (CRLF) to end lines, whereas unix/linux use just \n (LF). Since your file is being generated in Windows, the "blank" lines actually contain an invisible CR, and awk never sees a record separator.
To see the "real" contents of your file, you can use tools like hexdump or od. For example:
$ printf 'foo\r\nbar\r\n' | od -c
0000000 f o o \r \n b a r \r \n
0000012
In your case, simply run:
$ od -c filename | less
(Or use more if less isn't available.)
Many systems have a package available called dos2unix which can convert text format.
If you don't have dos2unix available, you should be able to achieve the same thing using other tools. In GNU sed:
sed -i 's/\r$//' filename
Or in other sed variants, but with a shell (like bash) that supports format substitution (read man sed to see if you have a -i option):
sed $'s/\r$//' inputfile > outputfile
Or a little less precisely, as it will remove all CRs even if they're not at the end of the line, you could use tr:
tr -d '\015' < inputfile > outputfile
Or if perl is available, you can use a substitution expression that's almost identical to the one for sed (perl documentation is readily available to tell you what the options do):
perl -i -pe 's/\r\n$/\n/g' filename
Good luck!