I have some applications, and standard Unix tools sending their output to named-pipes in Solaris, however named pipes can only be read from the local storage (on Solaris), so I can't access them from over the network or place the pipes on an NFS storage for networked access to their output.
Which got me wondering if there was an analogous way to forward the output of command-line tools directly to sockets, say something like:
mksocket mysocket:12345
vmstat 1 > mysocket 2>&1
Netcat is great for this. Here's a page with some common examples.
Usage for your case might look something like this:
Server listens for a connection, then sends output to it:
server$ my_script | nc -l 7777
Remote client connects to server on port 7777, receives data, saves to a log file:
client$ nc server 7777 >> /var/log/archive
netcat (also known as nc) is exactly what you're looking for. It's getting to be reasonably standard, but not available on all systems.
socat seems to be a beefed-up version of netcat, with lots more features, but less commonly available.
On Linux, you can also use /dev/tcp/<host>/<port>. See the Advanced Bash-Scripting Guide for more information.
netcat will help establish a pipe over the network.
You may want to use one of:
ssh: secure (encrypted), already installed out-of-the-box on Solaris - but you have to set up a keypair for non-interactive sessions
e.g. vmstat 2>&1 | ssh -i private.key oss#remote.node "cat >vmstat.out"
netcat: simple to set up - but insecure and open to attacks
see http://www.debian-administration.org/articles/58 etc.
Everyone is on the right track with netcat. But I want to add that if you are piping into nc and expecting a response, you will need to use the -q <seconds> option. From the manual:
-q seconds
after EOF on stdin, wait the specified number of seconds and then quit. If seconds is negative, wait forever.
For instance, if you want to interact with your SSH Agent you can do something like this:
echo -en '\x00\x00\x00\x01\x0b' | nc -q 1 -U $SSH_AUTH_SOCK | strings
A more complete example is at https://gist.github.com/RichardBronosky/514dbbcd20a9ed77661fc3db9d1f93e4
* I stole this from https://ptspts.blogspot.com/2010/06/how-to-use-ssh-agent-programmatically.html
Related
I am running a tunnel like this:
socat TCP-LISTEN:9090,fork TCP:192.168.1.3:9090
I would like to run a script to execute code with the strings passing through the tunnel.
The script does not change the strings, only processes strings independently but allows passage without changing between both ends.
Is this possible?
You should be able to even alter the communication using this approach:
Prepare a helper script helper.sh which gets executed for each connection:
#!/bin/bash
./inFilter.sh | socat - TCP:192.168.1.3:9090 | ./outFilter.sh
And start listening by using:
socat TCP-LISTEN:9090,fork EXEC:"./helper.sh"
The scripts inFilter.sh and outFilter.sh are processing the client and the server parts of the communication.
Example inFilter.sh:
#!/bin/bash
while read -r l ; do echo "IN(${l})" ; done
Example outFilter.sh:
#!/bin/bash
while read -r l ; do echo "OUT(${l})" ; done
This method should work for line-based text communication (as almost everything is line-buffered).
To make it work for binary protocols wrapping all processes with stdbuf -i0 -o0 might help (see .e.g here) and getting rid of shell is probably a good idea in this case.
Good luck!
I can run the following command to accomplish what I am trying to do, however I would like to setup entries in my ~/.ssh/config to handle a transparent jump:
ssh -tt login.domain.org gsissh -tt -p 2222 remote.behind.wall.domain.org
Note that the second hop MUST be made with gsissh, some info can be found here: http://toolkit.globus.org/toolkit/docs/5.0/5.0.4/security/openssh/pi/
AFAIK this precludes the standard use of netcat or the -W flag in the ProxyCommand option in the .ssh/config. I think this is because ssh will try to use ssh instead of gsissh on the intermediate machine.
If I put something like this in my .ssh/config it will hop through to the target machine, but when I exit I will land in a shell on the intermediate machine and it borks my ControlMaster setup—the next time I try to ssh to the final destination I end up on the intermediate machine
Host dest
HostName login.domain.org
PermitLocalCommand yes
LocalCommand gsissh -p 2222 remote.behind.wall.domain.org
Also, it seems that trickery using -L or -R is disabled for security reasons.
I would love some help if anybody has any tips.
Thanks
I'm trying to set up a pipeline of processing commands with unix pipes and FIFO:s (named pipes).
I also wanted to send/stream the output of the process to another compute node, which can start working on the stream of data as soon as it arrives, that is, I want to "pipe the stdout over to a process on another machine". How can I do that?
E.g. is it possible to set up a FIFO that will in the background write it's content over to a FIFO on the other compute node, or similar?
You can use netcat. In this minimal example, you can pipe the output of cat to netcat using:
cat local-filename.txt | netcat remote-hostname 1234
Where 1234 represents the TCP port that is going to be used. In the receiving side, you can use:
netcat -l 1234 > filename-on-remote-host.txt
where -l indicates you are setting up a server. This connection will be closed when the originating netcat process finishes. If you need it to keep going and waiting for the next connection, you can use the -k option:
netcat -kl 1234 | some-receiving-command
In any case you can use the abbreviated nc instead of the full netcat:
nc -kl 1234 | some-receiving-command
Yes it is possible, just use ssh for this purpose. The stdin of ssh is sent to the other host. You can use it for example to send data to a different server using tar:
tar cvzf - data | ssh otherhost 'cd /tmp; tar xvzf -'
I'm trying to plot the TCP congestion window and the slow start threshold using iperf and the tcp_probe module. I do exactly what is told here:
to obtain the data:
modprobe tcp_probe port=5001
chmod 444 /proc/net/tcpprobe
cat /proc/net/tcpprobe >/tmp/tcpprobe.out &
TCPCAP=$!
iperf -i 10 -t 100 -c receiver
kill $TCPCAP
Oops!
/tmp/tcpprobe.out is empty :(
This is Ubuntu 11.04 x86
and already tried the same on Ubuntu 11.04 x64
Any suggestions?
I was having the same problem. What worked for me was:
modprobe -r tcp_probe
sudo modprobe tcp_probe port=5002 full=1
sudo chmod 444 /proc/net/tcpprobe
cat /proc/net/tcpprobe > /tmp/tcpprobe.out &
TCPCAP=$!
iperf -c <servers IP address here> -p 5002 -t 100 -i 1
sudo kill $TCPCAP
See iperf parameters to check if those (-t 100 -i 1) are what you need by typing:
man iperf
I/O function in C standard library use buffer by default, usually 4k , so fread() only return when buffer full or EOF. You can use small buffer, 128 bytes, see:
dd if=/proc/net/tcpprobe ibs=128 obs=128
Now, message flush quickly.
By default the tcp_probe logs only when the cnwd changes, try modprobe tcp_probe ... full=1.
Linux source code referece: http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/net/ipv4/tcp_probe.c#L47
I had similiar issue, the tcp_probe module outputs only in non-obvious time intervals. I've created a modified version of it that flushes on every received tcp segments. This slows down the system, but allows to better monitor short-lived connections like HTTP.
Find the source code to the module here.
another issue which causes no output is file permission of output file tcpprobe.out
when cat tcpprobe directly, it's able to see the output, but if redirecting the output the file, the output file size is 0, which reminds me that it's the permission issue...
A very late answer, but have been struggling with this issue myself. I was trying out the version Dyna provided, yet still got no output, regardless of the parameters used. In the end, I found that the order was the problem.
The way I was using tcp_probe was: install/activate the module, run some tcp application (I was running some tcp unit tests), then start the copy process for /proc/net/tcpprobe (as shown in the other answers) and then remove/stop the module. The correct way is to start the copy process (barring killing of the process) BEFORE you perform the tcp intensive activity. Keep the cat process running while you perform the tcp activity and only kill it afterwards.
A pretty humbling experience for me, as it took hours to figure this out. Hopefully, people find this useful.
I am running a FreeBSD server and I have been sent a warning that spam has been sent from my server. I do not have it set as an open relay and I have customized the sendmail configuration. I'd like to know who is sending what email along with their username, email subject line as well as a summary of how much mail they have been sending. I would like to run a report on a log similar to how it is done when processing Apache server logs.
What are my options?
One idea is to alias sendmail to be a custom script, which simply cats the sendmail arguments to the end of a log before calling sendmail in the usual manner.
You can also monitor all system calls to write and read functions by executing:
ps auxw | grep sendmail | awk '{print"-p " $2}' | xargs strace -s 256 -f 2>&1 | grep -E $'#|(([0-9]+\.){3}[0-9]+)' | tee -a "/var/log/sendmail-logs.log"
This will give you direct access to the information, you cannot go deeper I think.
Can you give some sample logs? I think you're best bet would be to look through them with either grep or cut to get the source/destinations that are being sent too. Also, you could write a Perl script to automate it once you have the correct regex. This would be the best option.
If FreeBSD have default config, you have only one way to handle output mail, check what sending through you sendmail system in /etc/mail.
All output mail must be logged by /var/log/maillog