I'm trying to set up a pipeline of processing commands with unix pipes and FIFO:s (named pipes).
I also wanted to send/stream the output of the process to another compute node, which can start working on the stream of data as soon as it arrives, that is, I want to "pipe the stdout over to a process on another machine". How can I do that?
E.g. is it possible to set up a FIFO that will in the background write it's content over to a FIFO on the other compute node, or similar?
You can use netcat. In this minimal example, you can pipe the output of cat to netcat using:
cat local-filename.txt | netcat remote-hostname 1234
Where 1234 represents the TCP port that is going to be used. In the receiving side, you can use:
netcat -l 1234 > filename-on-remote-host.txt
where -l indicates you are setting up a server. This connection will be closed when the originating netcat process finishes. If you need it to keep going and waiting for the next connection, you can use the -k option:
netcat -kl 1234 | some-receiving-command
In any case you can use the abbreviated nc instead of the full netcat:
nc -kl 1234 | some-receiving-command
Yes it is possible, just use ssh for this purpose. The stdin of ssh is sent to the other host. You can use it for example to send data to a different server using tar:
tar cvzf - data | ssh otherhost 'cd /tmp; tar xvzf -'
Related
I'm trying to fetch few data from a Unix based server to my PC automatically ie I want the data to be transfered to my PC say every 30mins. I have the Unix code for fetching data but its through putty and it is getting stored in server only. I would like the data be stored in my local PC folder instead.
tail -n 10000 conveyor2.log | grep -P 'curing result OK' | sed 's/FT\/FT/g' | awk '{print $5 $13}' | uniq | sort -n | uniq >> my_data.txt
If you are currently using putty to connect to the server then you could also use "pscp" or "plink" on the Windows side to execute a transfer to your PC.
You would need to first understand how to do that from a command line.
For example:
pscp -i mykey.ppk user#serverName:logfileName targetName
(Using "-i mykey.ppk" allows you to bypass password prompts. You will need to create "mykey.ppk" using puttygen.)
You could then put that in a .BAT file or powershell or whatever and run it as a Windows "scheduled task" or get fancy and setup a service (which is well beyond the scope of this question).
For this first of all you can create a mount point of your PC on unix server.
this is called Samba.
Need root acess on both unix server and window machine
mount -t cifs //"ip-address of window system"/e$/ftp -o username="username",password="password" /"Mount point name"
after doing this you can directly create log file at window machine
I have a case when there are 2 remote hosts, each with a unique port number, which i both need to indicate when using SCP command. There is no problem indicating the unique (non 22) port number for the first one, but i can't figure out how to indicate it for the second one.
Here is how it looks like now -
scp -P 1234 username#site.com:/path/to/file username2#site2.com:/path/to/directory
The way that you are running scp in your question, it would just invoke the scp program on the first host, telling it to send the file to the second host. You can directly invoke ssh to run the scp command on the first host however you like it:
ssh -p1234 user1#host1 'scp -P2345 /path/to/file user2#host2:/path/to/directory'
If the first host can't connect directly to the second host, you'd normally use the scp option -3 to send the data from the first host to the second host through you local host. If you're in that position, you could emulate how scp runs that kind of transfer, though it's sort of a hack:
yes "" | tr '\n' '\0' |
ssh -p1234 user1#host1 'scp -f /path/to/file' |
ssh -p2345 user2#host2 'scp -t /path/to/directory'
This is based on how the SCP protocol works. The first ssh line runs scp on the first remote host in "remote source" mode to send the source file. The second ssh lines runs scp on the second host in "remote sink" mode to receive the file and write it to the directory. "remote source" and "remote sink" are what scp normally runs on the remote system when receiving or sending files, respectively.
The yes "" | tr '\n' '\0' | line creates a stream of NUL (Ascii 0) bytes which are needed for the protocol to work. The scp -f instance reads these and won't send files without them. There are other ways to generate a stream of NUL bytes if you like.
no idea how to do this with arguments, but you can try editing ~/.ssh/config
Host test
HostName test.com
Port 22000
User me
I'm piping from a RawCap-generated dump file to Wireshark in order to monitor local traffic, how can I instruct wireshark to only show traffic to a certain destination port?
I'm running RawCap in one Cygwin shell, and Wireshark in another to monitor RawCap's output:
Shell 1:
RawCap.exe -f 127.0.0.1 dumpfile.pcap
Shell 2:
# How do I tell Wireshark to show only traffic to port 10000?
tail -c +0 -f dumpfile.pcap | Wireshark.exe -k -i -
The appropriate flag for instructing wireshark to filter the displayed packets is -Y, as its man page reports:
-Y <display filter> start with the given display filter
For filtering the destination port of TCP, use tcp.dstport==X where X specifies the port.
Therefore, the full command is:
tail -c +0 -f dumpfile.pcap | wireshark -k -i - -Y "tcp.dstport==10000"
This is a good starting point for information on display filters. A full reference on the subject is available here and a detailed explanation of its syntax is available here. However, it's worth noting that most basic filters can be found via a simple online search.
I can run the following command to accomplish what I am trying to do, however I would like to setup entries in my ~/.ssh/config to handle a transparent jump:
ssh -tt login.domain.org gsissh -tt -p 2222 remote.behind.wall.domain.org
Note that the second hop MUST be made with gsissh, some info can be found here: http://toolkit.globus.org/toolkit/docs/5.0/5.0.4/security/openssh/pi/
AFAIK this precludes the standard use of netcat or the -W flag in the ProxyCommand option in the .ssh/config. I think this is because ssh will try to use ssh instead of gsissh on the intermediate machine.
If I put something like this in my .ssh/config it will hop through to the target machine, but when I exit I will land in a shell on the intermediate machine and it borks my ControlMaster setup—the next time I try to ssh to the final destination I end up on the intermediate machine
Host dest
HostName login.domain.org
PermitLocalCommand yes
LocalCommand gsissh -p 2222 remote.behind.wall.domain.org
Also, it seems that trickery using -L or -R is disabled for security reasons.
I would love some help if anybody has any tips.
Thanks
I have some applications, and standard Unix tools sending their output to named-pipes in Solaris, however named pipes can only be read from the local storage (on Solaris), so I can't access them from over the network or place the pipes on an NFS storage for networked access to their output.
Which got me wondering if there was an analogous way to forward the output of command-line tools directly to sockets, say something like:
mksocket mysocket:12345
vmstat 1 > mysocket 2>&1
Netcat is great for this. Here's a page with some common examples.
Usage for your case might look something like this:
Server listens for a connection, then sends output to it:
server$ my_script | nc -l 7777
Remote client connects to server on port 7777, receives data, saves to a log file:
client$ nc server 7777 >> /var/log/archive
netcat (also known as nc) is exactly what you're looking for. It's getting to be reasonably standard, but not available on all systems.
socat seems to be a beefed-up version of netcat, with lots more features, but less commonly available.
On Linux, you can also use /dev/tcp/<host>/<port>. See the Advanced Bash-Scripting Guide for more information.
netcat will help establish a pipe over the network.
You may want to use one of:
ssh: secure (encrypted), already installed out-of-the-box on Solaris - but you have to set up a keypair for non-interactive sessions
e.g. vmstat 2>&1 | ssh -i private.key oss#remote.node "cat >vmstat.out"
netcat: simple to set up - but insecure and open to attacks
see http://www.debian-administration.org/articles/58 etc.
Everyone is on the right track with netcat. But I want to add that if you are piping into nc and expecting a response, you will need to use the -q <seconds> option. From the manual:
-q seconds
after EOF on stdin, wait the specified number of seconds and then quit. If seconds is negative, wait forever.
For instance, if you want to interact with your SSH Agent you can do something like this:
echo -en '\x00\x00\x00\x01\x0b' | nc -q 1 -U $SSH_AUTH_SOCK | strings
A more complete example is at https://gist.github.com/RichardBronosky/514dbbcd20a9ed77661fc3db9d1f93e4
* I stole this from https://ptspts.blogspot.com/2010/06/how-to-use-ssh-agent-programmatically.html