I have a file One.lst that has the following content:
a
b
c
Now,i have to search in :
a_man.lst
b_man.lst
c_man.lst
the pattern : <inc>true</inc>.
Now,suppose b_man.xml has the above pattern.
So, I have to log b in another file, say Two.lst.
You can for example do:
while read line
do
grep -q "<inc>true</inc>" ${line}_man.lst && echo $line >> Two.lst
done < One.lst
The while read keeps reading the file names from One.lst and stores them in $line.
grep -q checks the existence of <inc>true</inc> in ${line}_man.lst. In case of match, it prints the file name ($line) into Two.lst.
I have two files System_Names & system_appendix_names . I want to Concatenate each & every line of one file to other file`s line and save output to other file.
root#bt:~/kevin/new# cat system_appendix_names
adm
-adm
_adm
root#bt:~/kevin/new#cat System_Names
help
not
now
give
you
haha
what
where
if
I made following script.
#!/usr/bin/env bash
cat System_Names | while read line1
do
cat system_appendix_names | while read line2
do
out="${line1}${line2}"
echo "$out" >> 1.txt
done
done
Output of script:
root#bt:~/kevin/new# cat 1.txt
helpadm
help-adm
help_adm
notadm
not-adm
not_adm
nowadm
now-adm
now_adm
giveadm
give-adm
give_adm
youadm
you-adm
you_adm
hahaadm
haha-adm
haha_adm
whatadm
what-adm
what_adm
whereadm
where-adm
where_adm
ifadm
if-adm
if_adm
Above script work for small amount of lines in file.Actually i have files which have more lines. SO i tried it, but can not Concatenate string line by line.
I have many files like ABC_Timestamp.txt , RAM_Timestamp.txthere timestamp will be different everytime. I want to copy this file into other directory but while copying I want append one string at the end of the file , so the format will be ABC_Timestamp.txt.OK and RAM_Timestamp.txt.OK. How to append the string in dynamic file. Please suggest.
My 2 pence:
(cat file.txt; echo "append a line"; date +"perhaps with a timestamp: %T") > file.txt.OK
Or more complete for your filenames:
while sleep 3;
do
for a in ABC RAM
do
(echo "appending one string at the end of the file" | cat ${a}_Timestamp.txt -) > ${a}_Timestamp.txt.OK
done
done
Execute this on command line.
ls -1|awk '/ABC_.*\.txt/||/RAM_.*\.txt/
{old=$0;
new="/new_dir/"old".OK";
system("cp "old" "new); }'
Taken from here
You can say:
for i in *.txt; do cp "${i}" targetdirectory/"${i}".OK ; done
or
for i in ABC_*.txt RAM_*.txt; do cp "${i}" targetdirectory/"${i}".OK ; done
How about first dumping the names of the file in another file and then moving file one by one.
find . -name "*.txt" >fileNames
while read line
do
newName="${line}appendText"
echo $newName
cp $line $newName
done < fileNames
I want to differentiate the STDOUT and STDERR messages in my terminal.
If a script or command is printing a message in terminal I want to differentiate by colors; is it possible?
(E.g. stderr font color is red, and stdout font color is blue.)
Example (using bold):
$date
Wed Jul 27 12:36:50 IST 2011
$datee
bash: datee: command not found
$alias ls
alias ls='ls --color=auto -F'
$aliass ls
bash: aliass: command not found
Create a function in a bash shell or script:
color()(set -o pipefail;"$#" 2>&1>&3|sed $'s,.*,\e[31m&\e[m,'>&2)3>&1
Use it like this:
$ color command -program -args
It will show the command's stderr in red.
Keep reading for an explanation of how it works. There are some interesting features demonstrated by this command.
color()... — Creates a bash function called color.
set -o pipefail — This is a shell option that preserves the error return code of a command whose output is piped into another command. This is done in a subshell, which is created by the parentheses, so as not to change the pipefail option in the outer shell.
"$#" — Executes the arguments to the function as a new command. "$#" is equivalent to "$1" "$2" ...
2>&1 — Redirects the stderr of the command to stdout so that it becomes sed's stdin.
>&3 — Shorthand for 1>&3, this redirects stdout to a new temporary file descriptor 3. 3 gets routed back into stdout later.
sed ... — Because of the redirects above, sed's stdin is the stderr of the executed command. Its function is to surround each line with color codes.
$'...' A bash construct that causes it to understand backslash-escaped characters
.* — Matches the entire line.
\e[31m — The ANSI escape sequence that causes the following characters to be red
& — The sed replace character that expands to the entire matched string (the entire line in this case).
\e[m — The ANSI escape sequence that resets the color.
>&2 — Shorthand for 1>&2, this redirects sed's stdout to stderr.
3>&1 — Redirects the temporary file descriptor 3 back into stdout.
Here's a hack that I thought of and it seems to work:
Given the following aliases for readability:
alias blue='echo -en "\033[36m"'
alias red='echo -en "\033[31m"'
alias formatOutput='while read line; do blue; echo $line; red; done'
Now, you need to first set the font color in your terminal to red (as the default, which will be used for stderr).
Then, run your command and pipe the stdout through formatOutput defined above (which simply prints each line as blue and then resets the font color to red):
shell$ red
shell$ ls / somenonexistingfile | formatOutput
The above command will print in both stderr and stdout and you'll see that the lines are coloured differently.
Hope this helps
UPDATE:
To make this reusable, I've put it all in a small script:
$ cat bin/run
#!/bin/bash
echo -en "\033[31m" ## red
eval $* | while read line; do
echo -en "\033[36m" ## blue
echo $line
echo -en "\033[31m" ## red
done
echo -en "\033[0m" ## reset color
Now you can use this with any command:
$ run yourCommand
I color stderr red by linking the file descriptor to a custom function that adds color to everything that goes through it. Add to following to your .bashrc:
export COLOR_RED="$(tput setaf 1)"
export COLOR_RESET="$(tput sgr0)"
exec 9>&2
exec 8> >(
perl -e '$|=1; while(sysread STDIN,$a,9999) {print
"$ENV{COLOR_RED}$a$ENV{COLOR_RESET}"}'
)
function undirect(){ exec 2>&9; }
function redirect(){ exec 2>&8; }
trap "redirect;" DEBUG
PROMPT_COMMAND='undirect;'
So what is happening? The debug trap is executed just before and immediately after executing a command. stderr is thus redirected before a command is executed to enable red output. PROMPT_COMMAND is evaluated before the prompt is shown and with this I restore stderr to its normal state. This is necessary because PS1 and PS2 (your prompt) are printed over stderr and I do not want a red prompt. voila, red output over stderr!
You should check out stderred: https://github.com/sickill/stderred
Yes it's not possible natively. You'll have to hack the tty management (in the kernel).
I somehow finished some little C wrapper before I saw the other answers :-)
Might be buggy, and values are hardcoded, don't use this except for testing.
#include "unistd.h"
#include "stdio.h"
#include <sys/select.h>
int main(int argc, char **argv)
{
char buf[1024];
int pout[2], perr[2];
pipe(pout); pipe(perr);
if (fork()!=0)
{
close(1); close(2);
dup2(pout[1],1); dup2(perr[1],2);
close(pout[1]); close(perr[1]);
execvp(argv[1], argv+1);
fprintf(stderr,"exec failed\n");
return 0;
}
close(pout[1]); close(perr[1]);
while (1)
{
fd_set fds;
FD_ZERO(&fds);
FD_SET(pout[0], &fds);
FD_SET(perr[0], &fds);
int max = pout[0] > perr[0] ? pout[0] : perr[0];
int v = select(max+1, &fds, NULL, NULL, NULL);
if (FD_ISSET(pout[0], &fds))
{
int r;
r = read(pout[0], buf, 1024);
if (!r) {close(pout[0]); continue;}
write(1, "\033[33m", 5);
write(1, buf, r);
write(1, "\033[0m", 4);
}
if (FD_ISSET(perr[0], &fds))
{
int r;
r = read(perr[0], buf, 1024);
if (!r) {close(perr[0]); continue;}
write(2, "\033[31m", 5);
write(2, buf, r);
write(2, "\033[0m", 4);
}
if (v <= 0) break;
}
return 0;
}
Edit: Compared to the shell solution, this one will preserve the order of lines/characters more often. (It's not possible to be as accurate as direct tty reading.) Hitting ^C won't show an ugly error message, and it behaves correctly on this example:
./c_color_script sh -c "while true; do (echo -n a; echo -n b 1>&2) done"
I'm surprised that nobody has actually figured out how to color stdio streams. This will color stderr red for the entire (sub)shell:
exec 3>&2
exec 2> >(sed -u 's/^\(.*\)$/'$'\e''[31m\1'$'\e''[m/' >&3)
In this case, &3 will hold the original stderr stream.
You should not be passing any commands to exec, only the redirects. This special case causes exec to replace the current (sub)shell's stdio streams with those that it receives.
There are a few caveats:
Since sed will be running persistently in a parallel subshell, any direct output immediately following a write to the colored stdio will probably beat sed to the tty.
This method uses a FIFO file descriptor; FIFO nodes only deal in lines. If you don't write a linefeed to the stream, your output will be buffered until a newline is encountered. This is not buffering on sed's part: it's how these file types function.
The most troublesome of the caveats is the first, but a race condition can be more or less avoided by applying similar processing to all outputs, even if you use the default color.
You can perform similar processing for single commands by piping to the same sed command with the normal pipe operator (|). Piped chains are executed synchronously, so no race condition will occur, though the last command in a pipe chain receives its own subshell by default.
Expanding on the answer #gospes gave, I added the functionality to print out partial lines without waiting for a newline, and some comments. Allows for better output from wget or typing in a interactive shell.
exec 9>&2
exec 8> >(
while [ "$r" != "1" ]; do
# read input, no field separators or backslash escaping, 1/20th second timeout
IFS='' read -rt 0.05 line
r=$?
# if we have input, print the color change control char and what input we have
if ! [ "${#line}" = "0" ]; then
echo -ne "\e[1;33m${line}"
fi
# end of line detected, print default color control char and newline
if [ "$r" = "0" ] ; then
echo -e "\e[0m"
fi
# slow infinite loops on unexpected returns - shouldn't happen
if ! [ "$r" = "0" ] && ! [ "$r" = "142" ]; then
sleep 0.05
fi
done
)
function undirect(){ exec 2>&9; }
function redirect(){ exec 2>&8; }
trap "redirect;" DEBUG
PROMPT_COMMAND='undirect;'
I used bold yellow (1;33) but you can replace it with whatever, red for example (31) or bold red (1;33), and I arbitrarily chose 0.05 seconds for re-checking for end-of-lines and pausing on unexpected return codes (never found any); it could probably be lowered, or possibly removed from the read command.
You can make use of grep for this. Note that this assumes that grep is configured to have coloured output (this is the default on many systems).
$ aliass ls 2> >(GREP_COLORS='ms=01;31' grep .) 1> >(GREP_COLORS='ms=01;32' grep .)
aliass: command not found
This is a little long winded, if you are simply wanting to distinguish stderr fromstdout you can simply do this:
$ (echo "this is stdout"; echo "this is stderr" >&2) | grep .
this is stderr
this is stdout
This will result in stdout being highlighted with the default grep colour and stderr being white.
This might be the opposite of what you want if your default grep colour is red. If so you can explicitly set the grep colour to green:
$ GREP_COLORS='ms=01;32'
$ (echo "this is stdout"; echo "this is stderr" >&2) | grep .
If you explicitly want to get red output for stderr:
$ GREP_COLORS='ms=01;31'
$ (echo "this is stdout"; echo "this is stderr" >&2) 2> >(grep .)
Note that this solution will not preserve the output order.
I would like to hear your directions on how to insert lines of header (all lines in a file) to another file (more bigger, several GB). I prefer the Unix/awk/sed ways of do that job.
# header I need to insert to another, they are in a file named "header".
##fileformat=VCFv4.0
##fileDate=20090805
##source=myImputationProgramV3.1
##reference=1000GenomesPilot-NCBI36
##phasing=partial
##INFO=<ID=NS,Number=1,Type=Integer,Description="Number of Samples With Data">
##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Depth">
##INFO=<ID=AF,Number=.,Type=Float,Description="Allele Frequency">
##INFO=<ID=AA,Number=1,Type=String,Description="Ancestral Allele">
##INFO=<ID=DB,Number=0,Type=Flag,Description="dbSNP membership, build 129">
##INFO=<ID=H2,Number=0,Type=Flag,Description="HapMap2 membership">
##FILTER=<ID=q10,Description="Quality below 10">
##FILTER=<ID=s50,Description="Less than 50% of samples have data">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">
##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Read Depth">
##FORMAT=<ID=HQ,Number=2,Type=Integer,Description="Haplotype Quality">
#CHROM POS ID REF ALT QUAL FILTER INFO
header="/name/of/file/containing/header"
for file in "$#"
do
cat "$header" "$file" > /tmp/xx.$$
mv /tmp/xx.$$ "$file"
done
You might prefer to locate the temporary file on the same file system as the file you are editing, but anything that requires inserting data at the front of the file is going to end up working very close to this. If you are going to be doing this all day, every day, you might assemble something a little slicker, but the chances are the savings will be minuscule (fractions of a second per file).
If you really, really must use sed, then I suppose you could use:
header="/name/of/file/containing/header"
for file in "$#"
do
sed -e "0r $header" "$file" > /tmp/xx.$$
mv /tmp/xx.$$ "$file"
done
The command reads the content of header 'after' line 0 (before line 1), and then everything else is passed through unchanged. This isn't as swift as cat though.
An analogous construct using awk is:
header="/name/of/file/containing/header"
for file in "$#"
do
awk '{print}' "$header" "$file" > /tmp/xx.$$
mv /tmp/xx.$$ "$file"
done
This simply prints each input line on the output; again, not as swift as cat.
One more advantage of cat over sed or awk; cat will work even if the big files are mainly binary data (it is oblivious to the content of the files). Both sed and awk are designed to handle data split into lines; while modern versions will probably handle even binary data fairly well, it is not what they are designed for.
I did it all with a Perl script, because I had to traverse a directory tree and handle various file types differently. The basic script was
#!perl -w
process_directory(".");
sub process_directory {
my $dir = shift;
opendir DIR, $dir or die "$dir: not a directory\n";
my #files = readdir DIR;
closedir DIR;
foreach(#files) {
next if(/^\./ or /bin/ or /obj/); # ignore some directories
if(-d "$dir/$_") {
process_directory("$dir/$_");
} else {
fix_file("$dir/$_");
}
}
}
sub fix_file {
my $file = shift;
open SRC, $file or die "Can't open $file\n";
my $file = "$file-f";
open FIX, ">$fix" or die "Can't open $fix\n";
print FIX <<EOT;
-- Text to insert
EOT
while(<SRC>) {
print FIX;
}
close SRC;
close FIX;
my $oldfile = $file;
$oldFile =~ s/(.*)\.\(\w+)$/$1-old.$2/;
if(rename $file, $oldFile) {
rename $fix, $file;
}
}
Share and enjoy! Or not -- I'm no Perl hacker, so this is probably double-plus-unoptimal Perl code. Still, it worked for me!