I use logger command to log messages to /var/log/messages But How do I use logger to save the standard out, error out messages? Something like this does not work.
grep `date +'%y%m%d'` /var/log/mysqld.log | sed 's/^/computer /' | logger 2> logger
You're confusing redirection to a process (pipes, or |) with redirection to a file (>).
You need to redirect stderr to stdout by using 2>&1, and then pipe (|) to your logger process.
e.g.
grep .... 2>&1 | logger
This assumes you're using sh or a variant. The syntax is different for csh. It's worth having a look at this excerpt from Unix Power Tools for more info (especially as your previous question seems strongly related).
Related
This is surprisingly hard to search for.
The only thing I can find, are TRAP* functions, which can be triggered via various signals.
But I really want to watch all stdout/stderr, and have a function trigger if a certain string is matched.
(example: refreshing kerberos credentials. A command fails and emits a standard error message indicating I need to authenticate. I want to automatically run the command to do so ;)
The shell doesn't see a command's stdout/stderr if not piped to the shell. So, you need to redirect stdout/stderr to your zsh function. But you can also send them to both your zsh function and somewhere else. For instance:
your_command 2>&1 | tee >(your_zsh_function)
or
your_command |& tee >(your_zsh_function)
or
your_command >>(your_zsh_function) >>/dev/tty 2>&1
your_zsh_function will grep its input for a string match. A drawback is that you may have buffering problems.
But concerning your example, if I understand correctly, you may want to use the expect utility: "programmed dialogue with interactive programs".
What is the difference between
cat dat | tee >(wc -l ) | some other command
and
cat dat | tee file | wc -l
in terms of what is happening under the hood?
I can understand the second one as tee is forking the stream into a file and also to a pipe. But I am confused with the first one.
The first notation is the process substitution of Bash 4.x (not in 3.x, or not all versions of 3.x).
As far as tee is concerned, it is given a file name (such as /dev/fd/64) to which it writes as well as to standard output; it is actually a file descriptor for the write end of a pipe. As far as wc is concerned, it reads its standard input (which is the read end of the pipe that is connected to /dev/fd/64 for tee), and writes its answer to the standard output of the shell invoking the pipeline (not the standard output of tee which goes down the pipeline).
Since >( is process substitiution of bash,
the first line says:
send the contents of file 'dat' into some other command
while process 'wc' is run with its input or output
connected to a pipe which also sends the content of 'dat'
check "Process Substitution" of bash manpage.
Usually, stdout is line-buffered. In other words, as long as your printf argument ends with a newline, you can expect the line to be printed instantly. This does not appear to hold when using a pipe to redirect to tee.
I have a C++ program, a, that outputs strings, always \n-terminated, to stdout.
When it is run by itself (./a), everything prints correctly and at the right time, as expected. However, if I pipe it to tee (./a | tee output.txt), it doesn't print anything until it quits, which defeats the purpose of using tee.
I know that I could fix it by adding a fflush(stdout) after each printing operation in the C++ program. But is there a cleaner, easier way? Is there a command I can run, for example, that would force stdout to be line-buffered, even when using a pipe?
you can try stdbuf
$ stdbuf --output=L ./a | tee output.txt
(big) part of the man page:
-i, --input=MODE adjust standard input stream buffering
-o, --output=MODE adjust standard output stream buffering
-e, --error=MODE adjust standard error stream buffering
If MODE is 'L' the corresponding stream will be line buffered.
This option is invalid with standard input.
If MODE is '0' the corresponding stream will be unbuffered.
Otherwise MODE is a number which may be followed by one of the following:
KB 1000, K 1024, MB 1000*1000, M 1024*1024, and so on for G, T, P, E, Z, Y.
In this case the corresponding stream will be fully buffered with the buffer
size set to MODE bytes.
keep this in mind, though:
NOTE: If COMMAND adjusts the buffering of its standard streams ('tee' does
for e.g.) then that will override corresponding settings changed by 'stdbuf'.
Also some filters (like 'dd' and 'cat' etc.) dont use streams for I/O,
and are thus unaffected by 'stdbuf' settings.
you are not running stdbuf on tee, you're running it on a, so this shouldn't affect you, unless you set the buffering of a's streams in a's source.
Also, stdbuf is not POSIX, but part of GNU-coreutils.
Try unbuffer (man page) which is part of the expect package. You may already have it on your system.
In your case you would use it like this:
unbuffer ./a | tee output.txt
The -p option is for pipeline mode where unbuffer reads from stdin and passes it to the command in the rest of the arguments.
You can use setlinebuf from stdio.h.
setlinebuf(stdout);
This should change the buffering to "line buffered".
If you need more flexibility you can use setvbuf.
You may also try to execute your command in a pseudo-terminal using the script command (which should enforce line-buffered output to the pipe)!
script -q /dev/null ./a | tee output.txt # Mac OS X, FreeBSD
script -c "./a" /dev/null | tee output.txt # Linux
Be aware the script command does not propagate back the exit status of the wrapped command.
The unbuffer command from the expect package at the #Paused until further notice answer did not worked for me the way it was presented.
Instead of using:
./a | unbuffer -p tee output.txt
I had to use:
unbuffer -p ./a | tee output.txt
(-p is for pipeline mode where unbuffer reads from stdin and passes it to the command in the rest of the arguments)
The expect package can be installed on:
MSYS2 with pacman -S expect
Mac OS with brew install expect
Update
I recently had buffering problems with python inside a shell script (when trying to append timestamp to its output). The fix was to pass -u flag to python this way:
run.sh with python -u script.py
unbuffer -p /bin/bash run.sh 2>&1 | tee /dev/tty | ts '[%Y-%m-%d %H:%M:%S]' >> somefile.txt
This command will put a timestamp on the output and send it to a file and stdout at the same time.
The ts program (timestamp) can be installed with the moreutils package.
Update 2
Recently, also had problems with grep buffering the output, when I used the argument grep --line-buffered on grep to it stop buffering the output.
If you use the C++ stream classes instead, every std::endl is an implicit flush. Using C-style printing, I think the method you suggested (fflush()) is the only way.
The best answer IMO is grep's --line-buffer option as stated here:
https://unix.stackexchange.com/a/53445/40003
This is my first question on stackoverflow!
I want to have a unix script that will run grep on the console output. Here is what my script does:
1. Telnet into a remote server (I have done this part successfully)
2. On successful login, the remote server displays outputs information on the console. I need to run grep on that console output (need help with this)
So, I need a script to run grep on the output appearing on the console.
Any thoughts??
Thanks,
Puneet
Use SSH instead. It's more secure and far easier to script.
ssh remoteusername#remotehost:/path/to/remote/script | grep 'something'
with appropriate key setup, it won't even prompt you for a password.
Have you tried I/O redirection? You could either do
$ your-command > output.txt
and then run grep on that file, or just directly pipe the output through grep like so
$ your-command | grep ...
See this article or google around for similar. There are probably thousands of good articles about this around the web.
Instead of telnet, I would suggest using netcat (nc). You could then pass your login credentials via standard input and grep the standard output (nc prints anything sent by the server on standard output).
nc <host> <port> <auth.txt | grep 'string'
What you want to do is probably using a pipe. You can probably see it in the above answers it's the | sign you see in the command. It may be difficult to locate on your keyboard, depending on the layout. (I have to admit it is not very often used).
Pipes will redirect the output of one command. Instead of sending it to the console, they will send it as an input of another command.
cmd1 | grep foo is equivalent to running grep foo on the output of cmd1 (you can replace cmd1 by your netstat command).
One last thing is that you can have as many pipes as you want. For instance on my machine I can run ls -ltr | tail -1 | awk '{print $9}' | grep foo to look for the word foo in the last modified file.
I am running a FreeBSD server and I have been sent a warning that spam has been sent from my server. I do not have it set as an open relay and I have customized the sendmail configuration. I'd like to know who is sending what email along with their username, email subject line as well as a summary of how much mail they have been sending. I would like to run a report on a log similar to how it is done when processing Apache server logs.
What are my options?
One idea is to alias sendmail to be a custom script, which simply cats the sendmail arguments to the end of a log before calling sendmail in the usual manner.
You can also monitor all system calls to write and read functions by executing:
ps auxw | grep sendmail | awk '{print"-p " $2}' | xargs strace -s 256 -f 2>&1 | grep -E $'#|(([0-9]+\.){3}[0-9]+)' | tee -a "/var/log/sendmail-logs.log"
This will give you direct access to the information, you cannot go deeper I think.
Can you give some sample logs? I think you're best bet would be to look through them with either grep or cut to get the source/destinations that are being sent too. Also, you could write a Perl script to automate it once you have the correct regex. This would be the best option.
If FreeBSD have default config, you have only one way to handle output mail, check what sending through you sendmail system in /etc/mail.
All output mail must be logged by /var/log/maillog