Control file based attribute population : Unix shell - unix

Question

I have a control file header.cntrl. it has details of header. Example below...
cat header.cntrl
id, name, age, location, phone number
Now I am getting files from different sources,
Source 1 is sending input.dat file in the following format
cat input.dat
id, name, age, location, status, phone number
1,Abc, 34,India, active, 9999999999
Source 2 is sending data in the following format
cat input_2.dat
id, age, name, qualification, status, phone number, location
2,24,xyz, L L B, Active, 88888-88888, India
So different sources are sending files in different formats. We would need to convert those input files to header.cntrl file format.
I was trying this using awk code, but for each source, I'll need to write an awk code. Can we do it with a single script which can be used for any new future source as well?

Answer 1

This reformat_data script can reformat the two "non-standard" input formats and any future source formats. The key idea is to use Perl hashes to store the appropriate headings and only print those that are needed as specified in the header.cntrl file.
cat $* | perl -ne '
BEGIN {
#std_header = ("id","name","age","location","phone number");
print join(",", #std_header), ",\n";
chomp($firstline=<>);
$firstline =~ s/,\s+/,/g;
#inputfile_header=split(/,/, $firstline);
%hash=();
}
chomp;
#row = split(/,/);
$i=0;
for $cell (#row) {
$cell =~ s/\s+//;
$header=$inputfile_header[$i];
$hash{$header} = $row[$i];
$i++;
}
foreach $cell (#std_header) {
print "$hash{$cell},";
}
print "\n";
'
Here are the results of running the reformat_data script using the two sample input files:
cat input.dat
id, name, age, location, status, phone number
1,Abc, 34,India, active, 9999999999
reformat_data input.dat
id,name,age,location,phone number,
1,Abc,34,India,9999999999,
cat input_2.dat
id, age, name, qualification, status, phone number, location
2,24,xyz, L L B, Active, 88888-88888, India
reformat_data input_2.dat
id,name,age,location,phone number,
2,xyz,24,India,88888-88888,

Answer 2

In this particular case you can check the number of fields in lines (provided that all lines of a file have the same number of fields) (awk code):
{
n = split($0, a, "[ \t]*,[ \t]*");
if (n < 7) {
print a[1] ", " a[2] ", " a[3] ", " a[4] ", " a[6];
}
else {
print a[1] ", " a[3] ", " a[2] ", " a[7] ", " a[6];
}
}
A more sophisticated solution is to use the first line as key identifier and take remaining fields "by name":
{
n = split($0, a, "[ \t]*,[ \t]*");
if (FNR == 1) {
for (i = 1; i <= n; ++i) {
lbl[a[i]] = i;
}
}
print a[lbl["id"]] ", " a[lbl["name"]] ", " a[lbl["age"]] ", " a[lbl["location"]] ", " a[lbl["phone number"]];
}