Related
I have a sed command that is capturing a single line with sometext. The line in the file it is capturing ends with a linefeed. I am trying to utilize this variable in a pipeline, however, when I attempt to echo, or use it with other commands requiring an input, the result is a blank. Ex:
sed '1,1!d' somefile.txt | echo "$1", I know the variable itself is not empty as I can replace echo "$1" with cat $1 and see the correct printout.
edit - I have tried piping to a tr -d and removing the newline. I have confirmed the newline character is gone, yet echos still show blank. Cats do not.
edit 2 - I piped the variable into an if statement ... | if [[ -z $1 ]]; then cat $1; fi it hits the if, is determined to be empty, so runs the cat, which prints a non-empty line to console. If the variable is empty why is cat still printing out information?
What is causing this inconsistency and how can I solve my problem? The ultimate goal is to run the output of one sed, through another to replace specific lines in a target file.
sed '1,1!d' somefile.txt | sed '2,1s/.*/'$1'/' targetfile.txt
Contents of somefile.txt:
these
are
words
Contents of targetfile.txt:
The next line should say these
This line should say these
The previous line should say these
Output of echo after sed:
<empty>
Output of cat after sed:
these
Output of 2nd sed, using input from 1st:
The next line should say these
the previous line should say these
You are confused about arguments and input data. Look at this:
$ echo "$1"
$ echo "foo" | if [[ -z $1 ]]; then cat $1; fi
foo
The first argument to my shell, $1 is empty so if [[ -z $1 ]] succeeds. The reason that cat $1 produces output is that you have a fundamental shell programming error in that statement - you aren't quoting your variable, $1. The correct syntax isn't cat $1, it's cat "$1". Look at the difference:
$ echo "foo" | if [[ -z $1 ]]; then cat "$1"; fi
cat: '': No such file or directory
We can simplify the code to make what's happening clearer:
$ echo "foo" | cat $1
foo
$ echo "foo" | cat "$1"
cat: '': No such file or directory
The reason that echo "foo" | cat $1 produces output is that the unquoted $1 is expanded by the shell to nothing before cat is called so that statement is equivalent to just echo "foo" | cat and so cat just copies the input coming in from the pipe to it's output.
On the other hand echo "foo" | cat "$1" generates an error because the shell expands "$1" to the null string before cat is called and so it's then asking cat to open a file named <null> and that of course does not exist, hence the error.
Always quote your shell variables unless you have a specific reason not to and fully understand all of the implications. Read a shell man page and/or google that if you're not sure what those implications are.
wrt another part of your code you have:
sed '1,1!d' somefile.txt | echo "$1"
but, unlike cat, echo neither reads it's input from a pipe nor from a file name passed as an argument. The input to echo is just the list of string arguments you provide it so while echo "foo" | cat will cause cat to read the input stream containing foo and output it, echo "foo" | echo will produce no output because echo isn't designed to read input from a pipe and so it'll just print a null string since you gave it no arguments.
It's not clear what you're really trying to accomplish but I think you might want to replace the 2nd line of targetfile.txt with the first line of somefile.txt. If so that's just:
awk '
NR==FNR { if (NR==1) new=$0; next }
FNR==2 { $0 = new }
{ print }
' somefile.txt targetfile.txt
Do not try to use sed to do it or you'll find yourself in escaping/quoting hell because, unlike awk, sed does not understand literal strings, see Is it possible to escape regex metacharacters reliably with sed.
You appear to want to extract the first line from file1 and use it to replace the second line in file2.
At the moment, you are extracting that value from the first file with your first sed but sending it to the second sed on its stdin rather than as a parameter ($1).
Your description is confusing so I will use this as file1:
File 1 Line 1
File 1 Line 2
File 1 Line 3
And this as file2:
File 2 Line 1
File 2 Line 2
File 2 Line 3
There are many ways to do this.
Method 1
# Extract line1 from file1
extracted=$(sed '1!d' file1)
# Replace line 2 in file2 with extracted value
sed "2s/.*/$extracted/" file2
Not sure why I feel like a dentist now :-)
If you want to put it all on one line, as some folks like to do:
x=$(sed '1!d' file1) && sed "2s/.*/$x/" file2
Method 2
This one is a bit tricky. It uses the first sed to write a script for the second sed:
sed 's/^/2s|.*|/;s/$/|/;q' file1 | sed -f /dev/stdin file2
If you look at the first sed on its own you will see it is generating a script for the second one:
sed 's/^/2s|.*|/;s/$/|/;q' file1
2s|.*|File 1 Line 1|
If you look at the second sed, you will see it is executing a script passed on its standard input:
sed -f /dev/stdin ...
Method 3
Easier still is awk:
awk 'FNR==NR{if(NR==1)saved=$0;next} FNR==2{$0=saved}1' file1 file2
File 2 Line 1
File 1 Line 1
File 2 Line 3
All you need to notice is that I am passing 2 files to awk and that FNR==NR inside the script means that awk is currently processing the first file, because FNR is the line number in the current file and NR is the total number of lines awk has processed from all files so far. So, when processing the second file, NR is greater than FNR by the number of lines in the first file.
Goal
In ZSH script, for a given args, I want to obtain the first string and the rest.
For instance, when the script is named test
sh test hello
supposed to extract h and ello.
ZSH manual
http://zsh.sourceforge.net/Doc/zsh_a4.pdf
says:
Subscripting may also be performed on non-array values, in which case the subscripts specify a
substring to be extracted. For example, if FOO is set to ‘foobar’, then ‘echo $FOO[2,5]’ prints
‘ooba’.
Q1
So, I wrote a shell script in a file named test
echo $1
echo $1[1,1]
terminal:
$ sh test hello
hello
hello[1,1]
the result fails. What's wrong with the code?
Q2
Also I don't know how to extract subString from n to the last. Perhaps do I have to use Array split by regex?
EDIT: Q3
This may be another question, so if it's proper to start new Thread, I will do so.
Thanks to #skishore Here is the further code
#! /bin/zsh
echo $1
ARG_FIRST=`echo $1 | cut -c1`
ARG_REST=`echo $1 | cut -c2-`
echo ARG_FIRST=$ARG_FIRST
echo ARG_REST=$ARG_REST
if $ARG_FIRST = ""; then
echo nullArgs
else
if $ARG_FIRST = "#"; then
echo #Args
else
echo regularArgs
fi
fi
I'm not sure how to compare string valuables to string, but for a given args hello
result:
command not found: h
What's wrong with the code?
EDIT2:
What I've found right
#! /bin/zsh
echo $1
ARG_FIRST=`echo $1 | cut -c1`
ARG_REST=`echo $1 | cut -c2-`
echo ARG_FIRST=$ARG_FIRST
echo ARG_REST=$ARG_REST
if [ $ARG_FIRST ]; then
if [ $ARG_FIRST = "#" ]; then
echo #Args
else
echo regularArgs
fi
else
echo nullArgs
fi
EDIT3:
As the result of whole, this is what I've done with this question.
https://github.com/kenokabe/GitSnapShot
GitSnapShot is a ZSH thin wrapper for Git commands for easier and simpler usage
A1
As others have said, you need to wrap it in curly braces. Also, use a command interpreter (#!...), mark the file as executable, and call it directly.
#!/bin/zsh
echo $1
echo ${1[1,1]}
A2
The easiest way to extract a substring from a parameter (zsh calls variables parameters) is to use parameter expansion. Using the square brackets tells zsh to treat the scalar (i.e. string) parameter as an array. For a single character, this makes sense. For the rest of the string, you can use the simpler ${parameter:start:length} notation instead. If you omit the :length part (as we will here), then it will give you the rest of the scalar.
File test:
#!/bin/zsh
echo ${1[1]}
echo ${1:1}
Terminal:
$ ./test Hello
H
ello
A3
As others have said, you need (preferably double) square brackets to test. Also, to test if a string is NULL use -z, and to test if it is not NULL use -n. You can just put a string in double brackets ([[ ... ]]), but it is preferable to make your intentions clear with -n.
if [[ -z "${ARG_FIRST}" ]]; then
...
fi
Also remove the space between #! and /bin/zsh.
And if you are checking for equality, use ==; if you are assigning a value, use =.
RE:EDIT2:
Declare all parameters to set the scope. If you do not, you may clobber or use a parameter inherited from the shell, which may cause unexpected behavior. Google's shell style guide is a good resource for stuff like this.
Use builtins over external commands.
Avoid backticks. Use $(...) instead.
Use single quotes when quoting a literal string. This prevents pattern matching.
Make use of elif or case to avoid nested ifs. case will be easier to read in your example here, but elif will probably be better for your actual code.
Using case:
#!/bin/zsh
typeset ARG_FIRST="${1[1]}"
typeset ARG_REST="${1:1}"
echo $1
echo 'ARG_FIRST='"${ARG_FIRST}"
echo 'ARG_REST='"${ARG_REST}"
case "${ARG_FIRST}" in
('') echo 'nullArgs' ;;
('#') echo '#Args' ;;
(*)
# Recommended formatting example with more than 1 sloc
echo 'regularArgs'
;;
esac
using elif:
#!/bin/zsh
typeset ARG_FIRST="${1[1]}"
typeset ARG_REST="${1:1}"
echo $1
echo 'ARG_FIRST='"${ARG_FIRST}"
echo 'ARG_REST='"${ARG_REST}"
if [[ -z "${ARG_FIRST}" ]]; then
echo nullArgs
elif [[ '#' == "${ARG_FIRST}" ]]; then
echo #Args
else
echo regularArgs
fi
RE:EDIT3
Use "$#" unless you really know what you are doing. Explanation.
You can use the cut command:
echo $1 | cut -c1
echo $1 | cut -c2-
Use $() to assign these values to variables:
ARG_FIRST=$(echo $1 | cut -c1)
ARG_REST=$(echo $1 | cut -c2-)
echo ARG_FIRST=$ARG_FIRST
echo ARG_REST=$ARG_REST
You can also replace $() with backticks, but the former is recommended and the latter is somewhat deprecated due to nesting issues.
So, I wrote a shell script in a file named test
$ sh test hello
This isn't a zsh script: you're calling it with sh, which is (almost certainly) bash. If you've got the shebang (#!/bin/zsh), you can make it executable (chmod +x <script>) and run it: ./script. Alternatively, you can run it with zsh <script>.
the result fails. What's wrong with the code?
You can wrap in braces:
echo ${1} # This'll work with or without the braces.
echo ${1[3,5]} # This works in the braces.
echo $1[3,5] # This doesn't work.
Running this: ./test-script hello gives:
./test-script.zsh hello
hello
llo
./test-script.zsh:5: no matches found: hello[3,5]
Also I don't know how to extract subString from n to the last. Perhaps do I have to use Array split by regex?
Use the [n,last] notation, but wrap in braces. We can determine how long our variable is with, then use the length:
# Store the length of $1 in LENGTH.
LENGTH=${#1}
echo ${1[2,${LENGTH}]} # Display from `2` to `LENGTH`.
This'll produce ello (prints from the 2nd to the last character of hello).
Script to play with:
#!/usr/local/bin/zsh
echo ${1} # Print the input
echo ${1[3,5]} # Print from 3rd->5th characters of input
LENGTH=${#1}
echo ${1[2,${LENGTH}]} # Print from 2nd -> last characters of input.
You can use the cut command:
But that would be using extra baggage - zsh is quite capable of doing all this on it's own without spawning multiple sub-shells for simplistic operations.
I have pipe-delimited text files that requires an MD5 hash of a particular field, or set of fields. Because I'm on AIX and have to use the csum function, I don't think I can simply pass the file and a hashing function to awk to do it in one fell swoop.
So I'm writing a script that reads through each line, passes the to-be-hashed field to csum, then drops the result back in as a replacement via a gsub. 99% of the time it seems to work OK, but sometimes something goes afoul because the gsub replaces something it shouldn't.
#!/bin/ksh
rm $2 #Get rid of output file
while read line; do #loop through each line
MYFIELD=$(echo "$line" | cut -d "|" -f 6); #push the 6th field into a var
MYHASH=$(echo $MYFIELD | csum -h MD5 -); #csum will hash a string only on the stdin
echo $line | sed -e "s/$MYFIELD/${MYHASH}/g" >> $2 #gsub replaces, but not always what we want
done < $1 #read in the input file
I think instead I could use awk to update the field. But it's beyond me how to do that one line at a time. Ideally I would like to have a script that would allow me to pass two mandatory parameters (infile and outfile) and then any number of field positions that would get hashed and replaced. A la
foo infile.txt outfile.txt 2 6 12
Which would read in infile.txt, hash fields 2, 6, and 12, and write out to outfile.txt.
Your suggestions would be most appreciated
What about doing it with awk?
Instead of
echo $line | sed -e "s/$MYFIELD/${MYHASH}/g" >> $2 #gsub replaces, but not always what we want
You can use
old=$MYFIELD; new=$MYHASH; echo $line | awk -F"|" -v o="$old" -v n="$new" '{OFS=FS} sub(o, n, $6) {print}' >> $2
Basically what we do is:
old=$MYFIELD; new=$MYHASH We assign the parameters to be sent to awk.
echo $line We output the line so that awk can get it.
In awk,
-F"|" define | as field separator.
-v o="$old" and -v n="$new" let awk work with variables $old and $new naming them o and n respectively.
{OFS=FS} - define the delimiter between fields. It could also be OFS="|", but this way we indicate awk to use the same we defined on -F="|". It is more flexible to keep the field separator in case it changes.
sub(o, n, $6) replaces the text on variable o (that is, $MYFIELD) with text on variable v (that is, $MYHASH), but just on field 6.
print the whole line with substituted text
This worked for me in the example you gave on comments:
old="hashit"; new="WE_DID"; echo "donthashit|foo1|bar1|foo2|bar2|hashit" | awk -F"|" -v o="$old" -v n="$new" '{OFS=FS} sub(o,n,$6) {print}'
donthashit|foo1|bar1|foo2|bar2|WE_DID
Hope it helps.
Edit
I found a way to pass variables to awk easily: -v o=${variable_name}
This way, the solution can be:
echo $line | awk -F"|" -v o=${MYFIELD} -v n=${MYHASH} '{OFS=FS} sub(o, n, $6) {print}' >> $2
Is there a Unix command to prepend some string data to a text file?
Something like:
prepend "to be prepended" text.txt
printf '%s\n%s\n' "to be prepended" "$(cat text.txt)" >text.txt
sed -i.old '1s;^;to be prepended;' inFile
-i writes the change in place and take a backup if any extension is given. (In this case, .old)
1s;^;to be prepended; substitutes the beginning of the first line by the given replacement string, using ; as a command delimiter.
Process Substitution
I'm surprised no one mentioned this.
cat <(echo "before") text.txt > newfile.txt
which is arguably more natural than the accepted answer (printing something and piping it into a substitution command is lexicographically counter-intuitive).
...and hijacking what ryan said above, with sponge you don't need a temporary file:
sudo apt-get install moreutils
<<(echo "to be prepended") < text.txt | sponge text.txt
EDIT: Looks like this doesn't work in Bourne Shell /bin/sh
Here String (zsh only)
Using a here-string - <<<, you can do:
<<< "to be prepended" < text.txt | sponge text.txt
This is one possibility:
(echo "to be prepended"; cat text.txt) > newfile.txt
you'll probably not easily get around an intermediate file.
Alternatives (can be cumbersome with shell escaping):
sed -i '0,/^/s//to be prepended/' text.txt
If it's acceptable to replace the input file:
Note:
Doing so may have unexpected side effects, notably potentially replacing a symlink with a regular file, ending up with different permissions on the file, and changing the file's creation (birth) date.
sed -i, as in Prince John Wesley's answer, tries to at least restore the original permissions, but the other limitations apply as well.
Here's a simple alternative that uses a temporary file (it avoids reading the whole input file into memory the way that shime's solution does):
{ printf 'to be prepended'; cat text.txt; } > tmp.txt && mv tmp.txt text.txt
Using a group command ({ ...; ...; }) is slightly more efficient than using a subshell ((...; ...)), as in 0xC0000022L's solution.
The advantages are:
It's easy to control whether the new text should be directly prepended to the first line or whether it should be inserted as new line(s) (simply append \n to the printf argument).
Unlike the sed solution, it works if the input file is empty (0 bytes).
The sed solution can be simplified if the intent is to prepend one or more whole lines to the existing content (assuming the input file is non-empty):
sed's i function inserts whole lines:
With GNU sed:
# Prepends 'to be prepended' *followed by a newline*, i.e. inserts a new line.
# To prepend multiple lines, use '\n' as part of the text.
# -i.old creates a backup of the input file with extension '.old'
sed -i.old '1 i\to be prepended' inFile
A portable variant that also works with macOS / BSD sed:
# Prepends 'to be prepended' *followed by a newline*
# To prepend multiple lines, escape the ends of intermediate
# lines with '\'
sed -i.old -e '1 i\
to be prepended' inFile
Note that the literal newline after the \ is required.
If the input file must be edited in place (preserving its inode with all its attributes):
Using the venerable ed POSIX utility:
Note:
ed invariably reads the input file as a whole into memory first.
To prepend directly to the first line (as with sed, this won't work if the input file is completely empty (0 bytes)):
ed -s text.txt <<EOF
1 s/^/to be prepended/
w
EOF
-s suppressed ed's status messages.
Note how the commands are provided to ed as a multi-line here-document (<<EOF\n...\nEOF), i.e., via stdin; by default string expansion is performed in such documents (shell variables are interpolated); quote the opening delimiter to suppress that (e.g., <<'EOF').
1 makes the 1st line the current line
function s performs a regex-based string substitution on the current line, as in sed; you may include literal newlines in the substitution text, but they must be \-escaped.
w writes the result back to the input file (for testing, replace w with ,p to only print the result, without modifying the input file).
To prepend one or more whole lines:
As with sed, the i function invariably adds a trailing newline to the text to be inserted.
ed -s text.txt <<EOF
0 i
line 1
line 2
.
w
EOF
0 i makes 0 (the beginning of the file) the current line and starts insert mode (i); note that line numbers are otherwise 1-based.
The following lines are the text to insert before the current line, terminated with . on its own line.
This will work to form the output. The - means standard input, which is provide via the pipe from echo.
echo -e "to be prepended \n another line" | cat - text.txt
To rewrite the file a temporary file is required as cannot pipe back into the input file.
echo "to be prepended" | cat - text.txt > text.txt.tmp
mv text.txt.tmp text.txt
Prefer Adam's answer
We can make it easier to use sponge. Now we don't need to create a temporary file and rename it by
echo -e "to be prepended \n another line" | cat - text.txt | sponge text.txt
Probably nothing built-in, but you could write your own pretty easily, like this:
#!/bin/bash
echo -n "$1" > /tmp/tmpfile.$$
cat "$2" >> /tmp/tmpfile.$$
mv /tmp/tmpfile.$$ "$2"
Something like that at least...
Editor's note:
This command will result in data loss if the input file happens to be larger than your system's pipeline buffer size, which is typically 64 KB nowadays. See the comments for details.
In some circumstances prepended text may available only from stdin.
Then this combination shall work.
echo "to be prepended" | cat - text.txt | tee text.txt
If you want to omit tee output, then append > /dev/null.
Another way using sed:
sed -i.old '1 {i to be prepended
}' inFile
If the line to be prepended is multiline:
sed -i.old '1 {i\
to be prepended\
multiline
}' inFile
Solution:
printf '%s\n%s' 'text to prepend' "$(cat file.txt)" > file.txt
Note that this is safe on all kind of inputs, because there are no expansions. For example, if you want to prepend !##$%^&*()ugly text\n\t\n, it will just work:
printf '%s\n%s' '!##$%^&*()ugly text\n\t\n' "$(cat file.txt)" > file.txt
The last part left for consideration is whitespace removal at end of file during command substitution "$(cat file.txt)". All work-arounds for this are relatively complex. If you want to preserve newlines at end of file.txt, see this: https://stackoverflow.com/a/22607352/1091436
As tested in Bash (in Ubuntu), if starting with a test file via;
echo "Original Line" > test_file.txt
you can execute;
echo "$(echo "New Line"; cat test_file.txt)" > test_file.txt
or, if the version of bash is too old for $(), you can use backticks;
echo "`echo "New Line"; cat test_file.txt`" > test_file.txt
and receive the following contents of "test_file.txt";
New Line
Original Line
No intermediary file, just bash/echo.
Another fairly straight forward solution is:
$ echo -e "string\n" $(cat file)
% echo blaha > blaha
% echo fizz > fizz
% cat blaha fizz > buzz
% cat buzz
blaha
fizz
You can do that easily with awk
cat text.txt|awk '{print "to be prepended"$0}'
It seems like the question is about prepending a string to the file not each line of the file, in this case as suggested by Tom Ekberg the following command should be used instead.
awk 'BEGIN{print "to be prepended"} {print $0}' text.txt
If you like vi/vim, this may be more your style.
printf '0i\n%s\n.\nwq\n' prepend-text | ed file
For future readers who want to append one or more lines of text (with variables or even subshell code) and keep it readable and formatted, you may enjoy this:
echo "Lonely string" > my-file.txt
Then run
cat <<EOF > my-file.txt
Hello, there!
$(cat my-file.txt)
EOF
Results of cat my-file.txt:
Hello, there!
Lonely string
This works because the read of my-file.txt happens first and in a subshell. I use this trick all the time to append important rules to config files in Docker containers rather than copy over entire config files.
you can use variables
Even though a bunsh of answers here work pretty well, I want to contribute this one-liner, just for completeness. At least it is easy to keep in mind and maybe contributes to some general understanding of bash for some people.
PREPEND="new line 1"; FILE="text.txt"; printf "${PREPEND}\n`cat $FILE`" > $FILE
In this snippe just replace text.txt with the textfile you want to prepend to and new line 1 with the text to prepend.
example
$ printf "old line 1\nold line 2" > text.txt
$ cat text.txt; echo ""
old line 1
old line 2
$ PREPEND="new line 1"; FILE="text.txt"; printf "${PREPEND}\n`cat $FILE`" > $FILE
$ cat text.txt; echo ""
new line 1
old line 1
old line 2
$
# create a file with content..
echo foo > /tmp/foo
# prepend a line containing "jim" to the file
sed -i "1s/^/jim\n/" /tmp/foo
# verify the content of the file has the new line prepened to it
cat /tmp/foo
I'd recommend defining a function and then importing and using that where needed.
prepend_to_file() {
file=$1
text=$2
if ! [[ -f $file ]] then
touch $file
fi
echo "$text" | cat - $file > $file.new
mv -f $file.new $file
}
Then use it like so:
prepend_to_file test.txt "This is first"
prepend_to_file test.txt "This is second"
Your file contents will then be:
This is second
This is first
I'm about to use this approach for implementing a change log updater.
With ex,
ex - $file << PREPEND
-1
i
prepended text
.
wq
PREPEND
The ex commands are
-1 Go to the very beginning of the file
i Begin insert mode
. End insert mode
wq Save (write) and quit
I ended up writing a quick little script for this in Python, but I was wondering if there was a utility you could feed text into which would prepend each line with some text -- in my specific case, a timestamp. Ideally, the use would be something like:
cat somefile.txt | prepend-timestamp
(Before you answer sed, I tried this:
cat somefile.txt | sed "s/^/`date`/"
But that only evaluates the date command once when sed is executed, so the same timestamp is incorrectly prepended to each line.)
ts from moreutils will prepend a timestamp to every line of input you give it. You can format it using strftime too.
$ echo 'foo bar baz' | ts
Mar 21 18:07:28 foo bar baz
$ echo 'blah blah blah' | ts '%F %T'
2012-03-21 18:07:30 blah blah blah
$
To install it:
sudo apt-get install moreutils
Could try using awk:
<command> | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'
You may need to make sure that <command> produces line buffered output, i.e. it flushes its output stream after each line; the timestamp awk adds will be the time that the end of the line appeared on its input pipe.
If awk shows errors, then try gawk instead.
annotate, available via that link or as annotate-output in the Debian devscripts package.
$ echo -e "a\nb\nc" > lines
$ annotate-output cat lines
17:00:47 I: Started cat lines
17:00:47 O: a
17:00:47 O: b
17:00:47 O: c
17:00:47 I: Finished with exitcode 0
Distilling the given answers to the simplest one possible:
unbuffer $COMMAND | ts
On Ubuntu, they come from the expect-dev and moreutils packages.
sudo apt-get install expect-dev moreutils
How about this?
cat somefile.txt | perl -pne 'print scalar(localtime()), " ";'
Judging from your desire to get live timestamps, maybe you want to do live updating on a log file or something? Maybe
tail -f /path/to/log | perl -pne 'print scalar(localtime()), " ";' > /path/to/log-with-timestamps
Kieron's answer is the best one so far. If you have problems because the first program is buffering its out you can use the unbuffer program:
unbuffer <command> | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; }'
It's installed by default on most linux systems. If you need to build it yourself it is part of the expect package
http://expect.nist.gov
Just gonna throw this out there: there are a pair of utilities in daemontools called tai64n and tai64nlocal that are made for prepending timestamps to log messages.
Example:
cat file | tai64n | tai64nlocal
Use the read(1) command to read one line at a time from standard input, then output the line prepended with the date in the format of your choosing using date(1).
$ cat timestamp
#!/bin/sh
while read line
do
echo `date` $line
done
$ cat somefile.txt | ./timestamp
I'm not an Unix guy, but I think you can use
gawk '{print strftime("%d/%m/%y",systime()) $0 }' < somefile.txt
#! /bin/sh
unbuffer "$#" | perl -e '
use Time::HiRes (gettimeofday);
while(<>) {
($s,$ms) = gettimeofday();
print $s . "." . $ms . " " . $_;
}'
$ cat somefile.txt | sed "s/^/`date`/"
you can do this (with gnu/sed):
$ some-command | sed "x;s/.*/date +%T/e;G;s/\n/ /g"
example:
$ { echo 'line1'; sleep 2; echo 'line2'; } | sed "x;s/.*/date +%T/e;G;s/\n/ /g"
20:24:22 line1
20:24:24 line2
of course, you can use other options of the program date. just replace date +%T with what you need.
Here's my awk solution (from a Windows/XP system with MKS Tools installed in the C:\bin directory). It is designed to add the current date and time in the form mm/dd hh:mm to the beginning of each line having fetched that timestamp from the system as each line is read. You could, of course, use the BEGIN pattern to fetch the timestamp once and add that timestamp to each record (all the same). I did this to tag a log file that was being generated to stdout with the timestamp at the time the log message was generated.
/"pattern"/ "C\:\\\\bin\\\\date '+%m/%d %R'" | getline timestamp;
print timestamp, $0;
where "pattern" is a string or regex (without the quotes) to be matched in the input line, and is optional if you wish to match all input lines.
This should work on Linux/UNIX systems as well, just get rid of the C\:\\bin\\ leaving the line
"date '+%m/%d %R'" | getline timestamp;
This, of course, assumes that the command "date" gets you to the standard Linux/UNIX date display/set command without specific path information (that is, your environment PATH variable is correctly configured).
Mixing some answers above from natevw and Frank Ch. Eigler.
It has milliseconds, performs better than calling a external date command each time and perl can be found in most of the servers.
tail -f log | perl -pne '
use Time::HiRes (gettimeofday);
use POSIX qw(strftime);
($s,$ms) = gettimeofday();
print strftime "%Y-%m-%dT%H:%M:%S+$ms ", gmtime($s);
'
Alternative version with flush and read in a loop:
tail -f log | perl -pne '
use Time::HiRes (gettimeofday); use POSIX qw(strftime);
$|=1;
while(<>) {
($s,$ms) = gettimeofday();
print strftime "%Y-%m-%dT%H:%M:%S+$ms $_", gmtime($s);
}'
caerwyn's answer can be run as a subroutine, which would prevent the new processes per line:
timestamp(){
while read line
do
echo `date` $line
done
}
echo testing 123 |timestamp
Disclaimer: the solution I am proposing is not a Unix built-in utility.
I faced a similar problem a few days ago. I did not like the syntax and limitations of the solutions above, so I quickly put together a program in Go to do the job for me.
You can check the tool here: preftime
There are prebuilt executables for Linux, MacOS, and Windows in the Releases section of the GitHub project.
The tool handles incomplete output lines and has (from my point of view) a more compact syntax.
<command> | preftime
It's not ideal, but I though I'd share it in case it helps someone.
The other answers mostly work, but have some drawbacks. In particular:
Many require installing a command not commonly found on linux systems, which may not be possible or convenient.
Since they use pipes, they don't put timestamps on stderr, and lose the exit status.
If you use multiple pipes for stderr and stdout, then some do not have atomic printing, leading to intermingled lines of output like [timestamp] [timestamp] stdout line \nstderr line
Buffering can cause problems, and unbuffer requires an extra dependency.
To solve (4), we can use stdbuf -i0 -o0 -e0 which is generally available on most linux systems (see How to make output of any shell command unbuffered?).
To solve (3), you just need to be careful to print the entire line at a time.
Bad: ruby -pe 'print Time.now.strftime(\"[%Y-%m-%d %H:%M:%S] \")' (Prints the timestamp, then prints the contents of $_.)
Good: ruby -pe '\$_ = Time.now.strftime(\"[%Y-%m-%d %H:%M:%S] \") + \$_' (Alters $_, then prints it.)
To solve (2), we need to use multiple pipes and save the exit status:
alias tslines-pipe="stdbuf -i0 -o0 ruby -pe '\$_ = Time.now.strftime(\"[%Y-%m-%d %H:%M:%S] \") + \$_'"
function tslines() (
stdbuf -o0 -e0 "$#" 2> >(tslines-pipe) > >(tslines-pipe)
status="$?"
exit $status
)
Then you can run a command with tslines some command --options.
This almost works, except sometimes one of the pipes takes slightly longer to exit and the tslines function has exited, so the next prompt has printed. For example, this command seems to print all the output after the prompt for the next line has appeared, which can be a bit confusing:
tslines bash -c '(for (( i=1; i<=20; i++ )); do echo stderr 1>&2; echo stdout; done)'
There needs to be some coordination method between the two pipe processes and the tslines function. There are presumably many ways to do this. One way I found is to have the pipes send some lines to a pipe that the main function can listen to, and only exit after it's received data from both pipe handlers. Putting that together:
alias tslines-pipe="stdbuf -i0 -o0 ruby -pe '\$_ = Time.now.strftime(\"[%Y-%m-%d %H:%M:%S] \") + \$_'"
function tslines() (
# Pick a random name for the pipe to prevent collisions.
pipe="/tmp/pipe-$RANDOM"
# Ensure the pipe gets deleted when the method exits.
trap "rm -f $pipe" EXIT
# Create the pipe. See https://www.linuxjournal.com/content/using-named-pipes-fifos-bash
mkfifo "$pipe"
# echo will block until the pipe is read.
stdbuf -o0 -e0 "$#" 2> >(tslines-pipe; echo "done" >> $pipe) > >(tslines-pipe; echo "done" >> $pipe)
status="$?"
# Wait until we've received data from both pipe commands before exiting.
linecount=0
while [[ $linecount -lt 2 ]]; do
read line
if [[ "$line" == "done" ]]; then
((linecount++))
fi
done < "$pipe"
exit $status
)
That synchronization mechanism feels a bit convoluted; hopefully there's a simpler way to do it.
doing it with date and tr and xargs on OSX:
alias predate="xargs -I{} sh -c 'date +\"%Y-%m-%d %H:%M:%S\" | tr \"\n\" \" \"; echo \"{}\"'"
<command> | predate
if you want milliseconds:
alias predate="xargs -I{} sh -c 'date +\"%Y-%m-%d %H:%M:%S.%3N\" | tr \"\n\" \" \"; echo \"{}\"'"
but note that on OSX, date doesn't give you the %N option, so you'll need to install gdate (brew install coreutils) and so finally arrive at this:
alias predate="xargs -I{} sh -c 'gdate +\"%Y-%m-%d %H:%M:%S.%3N\" | tr \"\n\" \" \"; echo \"{}\"'"
No need to specify all the parameters in strftime() unless you really want to customize the outputting format :
echo "abc 123 xyz\njan 765 feb" \
\
| gawk -Sbe 'BEGIN {_=strftime()" "} sub("^",_)'
Sat Apr 9 13:14:53 EDT 2022 abc 123 xyz
Sat Apr 9 13:14:53 EDT 2022 jan 765 feb
works the same if you have mawk 1.3.4. Even on awk-variants without the time features, a quick getline could emulate it :
echo "abc 123 xyz\njan 765 feb" \
\
| mawk2 'BEGIN { (__="date")|getline _;
close(__)
_=_" " } sub("^",_)'
Sat Apr 9 13:19:38 EDT 2022 abc 123 xyz
Sat Apr 9 13:19:38 EDT 2022 jan 765 feb
If you wanna skip all that getline and BEGIN { }, then something like this :
mawk2 'sub("^",_" ")' \_="$(date)"
If the value you are prepending is the same on every line, fire up emacs with the file, then:
Ctrl + <space>
at the beginning of the of the file (to mark that spot), then scroll down to the beginning of the last line (Alt + > will go to the end of file... which probably will involve the Shift key too, then Ctrl + a to go to the beginning of that line) and:
Ctrl + x r t
Which is the command to insert at the rectangle you just specified (a rectangle of 0 width).
2008-8-21 6:45PM <enter>
Or whatever you want to prepend... then you will see that text prepended to every line within the 0 width rectangle.
UPDATE: I just realized you don't want the SAME date, so this won't work... though you may be able to do this in emacs with a slightly more complicated custom macro, but still, this kind of rectangle editing is pretty nice to know about...