I found a couple of answers, but they seem to be specifically relating to Windows machines.
So my question is what are the differences between pipes and sockets, and when/how should you choose one over the other?
what are the differences between pipes and sockets, and when/how should you choose one over the other?
Both pipes and sockets handle byte streams, but they do it in different ways...
pipes only exist within a specific host, and they refer to buffering between virtual files, or connecting the output / input of processes within that host. There are no concepts of packets within pipes.
sockets packetize communication using IPv4 or IPv6; that communication can extend beyond localhost. Note that different endpoints of a socket can share the same IP address; however, they must listen on different TCP / UDP ports to do so.
Usage:
Use pipes:
when you want to read / write data as a file within a specific server. If you're using C, you read() and write() to a pipe.
when you want to connect the output of one process to the input of another process... see popen()
Use sockets to send data between different IPv4 / IPv6 endpoints. Very often, this happens between different hosts, but sockets could be used within the same host
BTW, you can use netcat or socat to join a socket to a pipe.
To complete the answer given by Mike, it is important to mention the existence of UNIX domain sockets, which are available on any POSIX compliant operating system. Although very similar to "normal" internet sockets in terms of usage semantics, they are purely local to the machine (of course internet sockets can also work locally), and thus almost behave like a pipe. Almost, because a UNIX pipe is by definition unidirectional:
Pipes and FIFOs (also known as named pipes) provide a unidirectional
interprocess communication channel. A pipe has a read end and a write
end. Data written to the write end of a pipe can be read from the read
end of the pipe. (excerpt from the man page pipe(7))
UNIX domain sockets also have a very unusual feature, as besides data, they also allow sending file descriptors: this way, an unprivileged process can access any file whose descriptor has been sent over the socket. This technique, according to Wikipedia, is used by the ClamAV antivirus scanning daemon.
Related
Among the following list of IPC options, which could perform multicast (i.e. 1 sender and multiple receivers):
signals
half duplex pipe
named pipe
system V message queue
unix domain socket
Edit
memory mapped files
From my understanding, it might be possible with named pipe (not sure).
There's nothing as conceptually flexible as multicast, but with a few limitations some of the facilities might do what you want.
Signals may be delivered to a process group. The other IPC mechanisms you list have a sender/receiver model and are not suitable for multicast, barring local extensions like Linux's multicast AF_UNIX sockets as #Barmar points out in the comments.
If you only need to send a single "signal" to descendant processes, and only once, you may use an inherited fifo. All receivers inherit the read end of the fifo but not the write end. The process holding the write end closes it at some point, and all receivers will detect EOF on their read end copies.
I wrote an server application in erlang and a client in C#. They communicate through 3 TCP ports. Port numbers are hardcoded. Now I'd like to do this dynamically. This is my first time doing network programming, so please pardon my inability to use proper terminology :-D
What I would like to do is make a supervisor which would accept a TCP connection from a client on a previously known port (say, 10000, or whatever), then find 3 free ports, start a server application on those 3 ports and tell the client those port numbers so client can connect to the server.
My particular problem is: how do I find 3 ports which are not in use? (clarification: which module:fun() to use to find a free port?)
My general problem is: I'm sure this kind of stuff (one server allocating ports and redirecting clients) is quite common network programming problem and there should be a bunch of (erlang-specific or general) resources about this, but I just don't have the terminology to google it out.
According to the Erlang documentation here, if the Port argument to the gen_tcp:listen/2 function is 0, then the OS will assign any available port to the socket. The latter can then be retreived using inet:port/1 .
You can therefore do something like this :
{ok, Listen} = gen_tcp:listen(0, [Options]),
Port = inet:port(Listen).
just in case you didn't know that - you dont have to allocate new ports for each client, it's perfectly fine to have all clients to connect to same ports
UPDATE:
if there is a reason to allocate new ports for incoming clients then it's far beyond your first "introduction to network programming" program.
separate ports could mean you want to completely isolate environments of different groups of clients. it's comparable to providing completely different IP addresses to connect to. if you want to write a simple ping-pong program - you don't need it. and i honestly believe you will never need to use such solution in your whole life - that's how incredibly rarely it is.
regarding cpu/ports overhead - allocating ports and starting a server that listens to that port is already far bigger overhead than accepting clients on same port.
You need to avoid commonly known ports, ftp, http, smtp etc, But I don't think there is any master list of which ports other software uses that you should avoid. I think your best bet is to come up with a range of ports you want to use. Check at runtime if anybody else answers ( is using the ports ) on the numbers you choose dynamically, if not issue it to the client.
I have a client which sends data via UDP-broadcast. (To let's say 127.0.0.255:12345)
Now I want to have multiple servers listening to this data. To do so on a local machine, they need to share the port 12345 for listening.
My question is, if that is possible, if there are any disadvantages and if there could be problems with this approach.
There is one alternative which unfortunately brings with a lot of overhead:
Implement some kind of registration-process. On startup, each server tells the client its port. The client then sends the messages to each port (having to send the data multiple times, some kind of handshaking needs to be implemented...)
Do you know any better alternative?
If that matters:
I'm using C++ with Boost::Asio. The software should be portable (mainly Linux and Windows).
You will have to bind the socket in both processes with the SO_REUSEPORT option. If you don't specify this option in the first process, binding in the second will fail. Likewise, if you specify this option in the first but not the second, binding in the second will fail. This option effectively specifies both a request ("I want to bind to this port even if it's already bound by another process") and a permission ("other processes may bind to this port too").
See section 4.12 of this document for more information.
This answer is referenced to the answer of cdhowie, who linked a document which states that SO_REUSEPORT would have the effect I'm trying to achieve.
I've researched how and if this option is implemented and focused mainly on Boost::Asio and Linux.
Boost::Asio does only set this option if the OS is equal to BSD or MacOSX. The code for that is contained in the file boost/asio/detail/reactive_socket_service.hpp (Boost Version 1.40, in newer versions, the code has been moved into other files).
I've wondered why Asio does not define this option for platforms like Linux and Windows.
There are several references discussing that this is not implemented in Linux:
https://web.archive.org/web/20120315052906/http://kerneltrap.org/mailarchive/linux-netdev/2008/8/7/2851754
http://kerneltrap.org/mailarchive/linux-kernel/2010/6/23/4586155
There also is a patch which should add this functionality to the kernel:
https://web-beta.archive.org/web/20110807043058/http://kerneltrap.org/mailarchive/linux-netdev/2010/4/19/6274993
I don't know if this option is existing for Windows, but by defining portable as an attribute for software which runs on Linux too, this means, that SO_REUSEPORT is OS specific and there is no portable solution for my question.
In one of the discussions I've linked it is recommended for UDP to implement a master-listener which then provides the incoming data to multiple slave-listeners.
I will mark this answer as accepted (though feeling kind of bad by accepting my own answer), because it points out why the approach of using SO_REUSEPORT will fail when trying to use it with portable software.
Several sources explain that you should use SO_REUSEADDR on windows. But none mention that it is possible to receive UDP message with and without binding the socket.
The code below binds the socket to a local listen_endpoint, that is essential, because without that you can and will still receive your UDP messages, but by default your will have exclusive ownership of the port.
However if you set reuse_address(true) on the socket (or on the acceptor when using TCP), and bind the socket afterwards, it will enable multiple applications, or multiple instances of your own application to do it again, and everyone will receive all messages.
// Create the socket so that multiple may be bound to the same address.
boost::asio::ip::udp::endpoint listen_endpoint(
listen_address, multicast_port);
// == important part ==
socket_.open(listen_endpoint.protocol());
socket_.set_option(boost::asio::ip::udp::socket::reuse_address(true));
socket_.bind(listen_endpoint);
// == important part ==
boost::array<char, 2000> recvBuffer;
socket_.async_receive_from(boost::asio::buffer(recvBuffer), m_remote_endpoint,
boost::bind(&SocketReader::ReceiveUDPMessage, this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred)
Just a technical question -
Can two or more SNMP agents be run on the same port (on the same machine)?
My first instinct would be no since host:port identifies an instance of an application but I'm not sure.
Thank you!
Technically, if the OS supports it, the SO_REUSEADDR SO_REUSEPORT options may be set on a socket to allow other processes to bind to the same address/port and thus allow multiple processes to receive messages on the same address/port. But both processes would have to set the option, and I doubt any agent implementations do that because it would not make sense to do so--it would just cause headaches having both agents potentially responding to a single request. Managers won't be equipped to handle it.
However, you can instead run an SNMP proxy in the primary address/port, configured to forward requests to one of multiple agents based on query, security, or (with SNMPv3) context/engine ID parameters, and forward responses back.
Also, using AgentX, you have an SNMP master agent running on the primary address/port, and one or more SNMP sub-agents connected to the master agent. The master agent dispatches requests to the sub-agents as appropriate, merging the results into a single response, so that to the outside world it appears as a single agent. Each sub-agent typically handles a different branch of OID space (one sub-agent implementing certain module(s), another sub-agent implementing other module(s)).
But taking two agents intended to own the address/port exclusively, and forcing them to share through the REUSE options, while it may be possible, would not be wise.
You can run multiple agents on the same host and with the same port if they have differents ip address (can use a netsh script for that).
Personnaly I use the nsoftware ddl : SecureSNMP V8 edition .NET to do this.
You can look at this post : Multiple SNMP Agents with nsoftware dll
No, two agents cannot both run on the same port as seperate applications for the reasons you assumed (except with a brittle packet sniffing hack, which we'll not go into).
However, 2 agents can be accessed through the same port if there is some mechanism that handles the actual port and distributes requests based on MIB. For example the Windows SNMP service does this, allowing any number of SNMP agents to be added as "extensions" through the registry (HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SNMP\Parameters\ExtensionAgents) by writing them as DLLs and using the snmp.h headers in the platform SDK.
You are correct: ports can't be shared.
If both the agents were designed by you, then the answer can be different.
Consider the HTTP and FTP cases, we can use host names to distinguise multiple sites on the same port, then why can't we do it for SNMP?
We can create a dispatcher who monitors port 161 for incoming traffic. Then use multiple real agents to handle those traffic behind. We can feel free to design how to distinguise them. Personally I prefer the FTP virtual host name manner and use | to distinguise agents.
Maybe I can create a demo for #SNMP Suite in the future.
But if you need to work with existing agents on the same server, then such flexibility is lost.
I am considering using SCTP instead of TCP for a p2p app written in C. Should I do it? Also how does the speed of SCTP compare to the speed of TCP?
EDIT:
I found that SCTP can be tunneled over UDP with the only problem being tunneled SCTP is not interoperable with untunneled SCTP.
Have you considered whether your target systems will all have SCTP pre-installed on them or whether your application will need to include SCTP itself? In my experience I would not expect all systems to have SCTP installed on them, and I would expect them not to if it were Windows.
If you include SCTP in the application itself then that will more than double the number of messages being passed into an out of the Kernel which will impact performance when compared with using the pre installed TCP.
Have you considered what benefits you want from SCTP? You mentioned fault tolerance but for this to work with SCTP it requires the application to have multiple ethernet ports and and IP addresses. Is this likely on your app?
As much as I love SCTP (!) I would seriously consider sticking with TCP unless you are sure SCTP is needed or unless you control the hosts your app is deployed on.
Regards
If it's for a local area network, sure go for it.
Note however that if you plan to use it on the open internet many consumer grade firewalls aren't flexible enough to permit unrecognised IP protocols through them.
How does it help you?
You're P2P, so every peer must have at least one socket open to every other peer.
If you've got a socket open, then you can do everything you need to do over that. If you've taken the approach of one socket per file and you have multiple files being tranferred concurrently between two given peers, then SCTP will save you one socket per file. However, on a normal P2P network of any size, you will almost never have multiple files being transferred concurrently between two peers.
Just have one socket and have your own little protocol; send a packet with a header, the header indicates content type, e.g. a command, or part a file - and if so, which file, and which byte range.
Of course, you get a little overhead for that, whereas if you have one socket for commands and one per file, you're more efficient. Is saving one socket per peer (assuming one download at a time) worth the time/hassle/complexity of using SCTP?