I'm debugging an IOT protocol between two essentially black boxes on my local network talking over a telnet connection. On either end, I can specify the IP address and port. I'd like to observe / record the data exchanged between the client and server.
I'm thinking a proxy running on my Mac laptop might work. I've done some research and experimentation with nc (BSD netcat). I can only figure out a working one-way pipe, and thus the protocol exchange does not happen between A and B.
Telnet TCP server (A) <===============> TCP Client (B)
Telnet TCP server (A) <===\ /==> TCP Client (B)
\ /
\ /
Proxy/Intercept (C) *
Using a feature in the server device (A), I can have two telnet connections active. Using this, I've been able to see the server's (A) responses to whatever Client (B) is commanding, but I cannot see the Client (B) commands. ~80% of the responses are a code meaning invalid command received, but a few are reasonable responses for what this thing should be doing. I can also note that that data rate is only about 4 Bytes / second, so I'd be happy to just watch this exchange live in a terminal.
To clarify: I can power-cycle the client (B), and it will re-establish a socket connection to the IP address and port of my choosing, so I'm not really thinking about a sniffing / Wireshark type solution.
I guess I'm hoping for a relatively straightforward solution run in the shell using existing standard tools. I suppose a small program could be written in Python or something to do this, but I hope that's overkill.
Related
I am puzzled...
My project is a PC connected to multiple micro-controller boards in an isolated network. So far the protocol has been UDP which is easy to deal with, has no particular client/server but has its obvious shortcomings of lost messages when things get busy.
The micro-controllers have fixed IP-addresses (set by dip switches), the PC SW has a list of them, sends at present UDP messages to each of them and they reply to the address they came from (i.e. the PC) with status and/or data.
My question is now that I switch to TCP instead of UDP, should the PC be the listening server with many clients (could be anything from 1 - 50), or should the micro controllers be listening servers the PC can connect to as client? Note: controllers have fixed/known addresses - the PC does not.
An additional concern is re-connection. The micro-controllers are external and may lose connection, reset or otherwise need to connect again.
Thanks....
should the PC be the listening server with many clients (could be anything from 1 - 50), or should the micro controllers be listening servers the PC can connect to as client?
That is a basic design question that we cannot answer for you. Likely, it's more practical for arbitrary devices to connect to a central server but that's not a given.
controllers have fixed/known addresses - the PC does not.
That might turn the previous question around.
The micro-controllers are external and may lose connection, reset or otherwise need to connect again.
That's something you need to put into your design - have TCP connections time out and reconnect. Usually, a finite-state machine is useful here. You should also consider whether you use a one-shot connect-transmit-disconnect similar to UDP (easier to implement) or a longer TCP session with multiple data transmissions (more efficient).
I am an embedded software developer who has any experience with TCPIP on connected devices. Also, I am not a software protocol expert, so I am a bit confusing about TCPIP protocol stack + responsiblities of its various phy layers.
First of all, I have experiences with such protocols like UART, SPI, CAN, USB... As you know, the phy layer directly affects you while selecting the protocol you used at the software level. For example, if you use usb and you build a software protocol on it, you do not occasionally deal with some details like checking corrupted frame in your sofware protocol, because phy layer of it guarantees this operation. CAN also has some CAN Controller facilities like crc and bit stuffing so, it is really reliable. But the situation is not the same for simple peripherals like UART/USART. Let's say you are using a bluetooth module to upgrade your firmware, you need to be aware of almost everything that can occur while communicating like delays, corrupted frames, payload validating etc.
Briefly, i am trying to understand the exact role of newtork interfaces come included in MCUs, that are interfaced with RJ45 phy sockets directly. In another words, imagine that I wrote a server application on my pc. Also i configured and ran an application in my development board which has an RJ45 socket and it runs as a client. Also imagine they established a connection over TCP. So, what will be the situation at the client side, when i send a 32 bytes of data to the socket from the server side? What will I see at the lowest level of MCU that is an RxCompleteInterrupt()? Are the data I sent and some other stuffs appended to the TCP packet guaranteed to be delivered by the eth controller in the MCU and ethernet controller of my PC? OR am i responsible (or the stack i used) check all the things necessary to validate whether the frame is valid or not?
I tried to be as clear as it would be. Please if you have experience, then try to write clean comments. I am not a TCPIP expert, maybe I used some wrong terminology, please focus the main concept of the question.
Thanks folks.
If you don't have any prior experience with the TCP/IP protocol suite, I would strongly suggest you to have a look at this IBM Redbook, more specifically at chapters 2, 3 and 4.
This being said:
So, what will be the situation at the client side, when i send a 32
bytes of data to the socket from the server side? What will I see at
the lowest level of MCU that is an RxCompleteInterrupt()?
You should have received an Ethernet frame in your buffer. This Ethernet frame should contain an IP packet. This IP packet should contain a TCP packet, which payload should consist in your 32 bytes of data. But there will be several exchanges between the client and the server prior to your data to be received, because of TCP being a connection-oriented protocol, i.e. several Ethernet frames will be sent/received.
Are the data I sent and some other stuffs appended to the TCP packet
guaranteed to be delivered by the eth controller in the MCU and
ethernet controller of my PC? OR am i responsible (or the stack i
used) check all the things necessary to validate whether the frame
is valid or not?
The TCP packet will ultimately be delivered, but there there are not warranties that your Ethernet frames and IP packets will be delivered, and will arrive in the right order. This is precisely the job of TCP, as a connection-oriented protocol, than to do what is needed so that the data you are sending as a TCP payload will ultimately be delivered. Your MCU hardware should be the one responsible for validating the Ethernet frames, but the TCP/IP stack running on the MCU is responsible for validating IP and TCP packets and the proper delivery of the data being sent/received over TCP.
You can experiment with TCP on a Linux PC using netcat, and capture the exchange using Wireshark or tcpdump.
Create a 'response' file containing 32 bytes:
echo 0123456789ABCDEFGHIJKL > response.txt
Start Wireshark, and filter on lo interface using filter tcp port 1234
Start a TCP server listening on TCP port 1234, which will send the content of response.txt upon receiving a connection from the client:
netcat -l 1234 < response.txt
In another console/shell, connect to the server listening on tcp/1234, and display what was received:
netcat localhost 1234
0123456789ABCDEFGHIJKL
On Wireshark, you should see the following Wireshark Network Capture, and be able to expand all frames/packets of the full exchange using the IBM Redbook as a reference.
Your 32 bytes of data will be in the payload section of a TCP packet sent by the server.
Ok so when I tried to do research on ip addresses, ports, and sockets, this is what I got out of it:
IP Addresses are used to map to different devices over a network.
Port numbers are used to get to the specific application on the hosts.
Sockets are a combination of the two..
What I don't understand is that if ports connect you to a specific application, you should only have 1 port number per application right? But for example port 80 is used for HTTP, so if an application is using that port it's listening to HTTP requests right? So what happens if more than one person tries to access it? Sockets and ports have me confused a lot..
A socket is an abstraction used in software to make it easier for programmers to send and receive data through networks. They are an interface, which you use in application-level code, to access the underlying network protocol implementations provided by your OS and language runtime.
The TCP protocol, IP protocol, and other popular network protocols do not, in of themselves, have any concept of "sockets". "Sockets" are a concept which implementers of TCP/IP came up with.
So what is the concept of a "socket"? Basically, an object which you can write data to, and read data from. "Opening" a socket means creating one of those objects in your program's memory. You can also "close" a socket, which means freeing any system resources which that object uses behind the scenes.
Some kinds of sockets can be "bound" to local and remote addresses, which you can think of as setting some data fields, or properties, on the socket object. The value of those fields affect what happens when you read from or write to the socket.
In Unix, there are various kinds of sockets. If you "open" a TCP socket, "bind" it to local and remote addresses (and ports), and write some data into it, your libraries/OS will package that data up into a TCP segment and send it out through whichever network interface matches the local address which you "bound" the socket to. If you "open" an IP socket, and write some data to it, that data will be packaged up into a IP packet (without any added TCP headers) and sent out. If you open a "raw", link-level socket, and write to it, the data will be sent out as the payload of a link-level frame, minus IP and TCP headers. There are also "Unix domain sockets". If you open one of those and write to it, the data will be passed directly through system memory to another process on the same machine.
So although they are often used in non-OO languages like C, sockets are a perfect example of what OO languages call "polymorphism". If you ever have trouble explaining what "polymorphism" is to someone, just teach them about network sockets.
"Ports" are a completely different concept. The idea of "ports" is built in to TCP and other transport protocols.
Others may give more high-falutin', and perhaps more technically accurate, definitions of a "port". Here is one which is totally down to earth:
A "port" is a number which appears in the TCP headers on a TCP segment. (Or the UDP headers on a UDP segment.)
Just a number. Nothing more, nothing less.
If you are using a "socket"-based interface to do network programming, the significance of that number is that each of your TCP or UDP sockets has a "local port" property, and a "remote port" property. As I said before, setting those properties is called "binding".
If your socket's "local port" property is "bound" to 80, then all the TCP segments you send out will have "80" in the "sender port" header. Then, when others respond to your messages, they will put "80" in their "destination port" headers.
More than that, if your socket is "bound" to local port 80, then when data arrives from elsewhere, addressed to your port 80, the OS will pass it to your application process and not any other. Then, when you try to read from the socket, that data will be returned.
Obviously, the OS needs to know what port each of your sockets is bound to. So when "binding", system calls must be made. If your program is not running with sufficient privileges, the OS may refuse to let you bind to a certain port. Then, depending on the language you are using, your networking library will throw an exception, or return an error code.
Sometimes the OS may refuse to let you bind to a certain port, not because you don't have the right privileges, but because another process has already bound to it. However, and this is what some of the other answers get wrong, if certain flags are set when opening a socket, your OS may allow more than one socket to be bound to the same local address and port.
You still don't know what "listening" and "connected" sockets are. But once you understand the above, that will just be a small jump.
The above explains the difference between what we today call a "socket" and what we call a "port". What may still not be clear is: why do we need to make that distinction?
You have really got me thinking here (thank you)! Could we call the software abstraction which is called a "socket" a "port" instead, so that instead of calling socket_recv you would call port_recv?
If you are only interested in TCP and UDP, maybe that would work. Remember, the "socket" abstraction is not only for TCP and UDP. It is also for other network protocols, as well as for inter-process communication on the same machine.
Then again, a TCP socket does not only map to a port. A "connected" TCP socket maps to a local IP address, local port, remote address, and remote port. It also has other associated data, including various flags, send and receive buffers, sequence numbers for the incoming/outgoing data streams, and various other variables used for congestion control (rate limiting), etc. That data does not belong just to a local port.
There can be thousands of TCP connections going simultaneously through the same "port". Each of those connections has its own associated data, and the software object which encapsulates that per-connection data is a "TCP socket".
Even if you only use TCP/UDP, and even if you only have a single process using any given local port at one time, and even if you only have a single connection going through each local port at one time, I think the "socket" abstraction still makes sense. If we just called sockets "ports", there would be more meanings conflated in that one word. Reusing the same word for too many meanings hinders communication.
"Ports" are transport-protocol level identifiers for an application process. "Sockets" are the objects used in software to send/receive messages which are addressed from/to those identifiers.
Differentiating between "my address" and "the thing which sends letters addressed as coming from me" is a useful distinction to make. "My address" is just a label. A label is not something active, which does things like sending data. It is logical to give "the thing which is used to send data" its own name, different from the name which denotes "the sender address which the data is labelled with".
When application (say web server like Apache or Nginx) is listening on say port 80, it creates so called listening socket.
When some client comes, this listening socket gets update (which can be noticed via select or poll API), and our application creates communication socket. This socket is uniquely identified by tuple (src_addr, src_port, dst_addr, dst_port) - it is very much possible that many clients will have exact same (dst_addr, dst_port) combination.
Then our web server can talk over that communication socket to deliver say web page and eventually close this socket. When many clients come in parallel, web server can either create thread/process per client (Apache model), or service all sockets one by one (Nginx model).
Note that in this situation only one listening socket per port can exist - multiple application cannot bind to the same port like 80. But, it is perfectly ok to have many communication sockets (some people report successfully serving more than a million simultaneous requests).
Every time you accept a connection on a socket in listening state (e.g. on port 80), you will get a new socket in established state that represents a connection.
On the client side, each time a new connection (new socket that is being connected) is being made with that address and port, the operating system will assign a random port on your side.
For example if you connect two times:
your-host:22482 <---> remote-host:80
your-host:23366 <---> remote-host:80
I have UDP network traffic arriving on my machine (OSC traffic from an iPad, to be exact) and I want two programs to be able to receive that traffic. The problem is that I can't bind to the same network port with two programs at once and I can't send to multiple ports with the iOS app I'm using. How can I solve this problem?
You can use the power of the command line for this. The following snippet uses socat (probably needs to be installed beforehand) and tee (should be preinstalled on any OS X or Linux).
socat -u UDP4-RECVFROM:8123,fork - | tee >(socat -u - UDP4-SENDTO:localhost:8223) | socat -u - UDP4-SENDTO:localhost:8323
Explanation: socat listens for traffic on UDP port 8123, pipes it to tee, which pipes it to two other instances of socat forwarding it to ports 8223 and 8323 on localhost respectively. With your two programs you need to listen to those ports on localhost.
While the answer with using socat is elegant it is not clear for me, what you are trying to do:
both programs should receive all parts of the traffic and they will only receive and not reply. This can be done with the proposed socat way
both program should receive all parts of the traffic and there reply will be mixed together (how?)
each of the programs should only receive parts of the traffic, e.g. the one which the other did not get. This should be possible if both of your programs use SO_REUSEADDR, SO_REUSEPORT. Replies will then be mixed together.
or do you actually want to communicate with each of the programs seperatly - then you would have to use either multiple sockets in the iOS app (which you don't want to do) or built your own protocol which does multiplexing, e.g. each message is prefixed with there target app and on the target machine a demultiplexer application will receive all packets and forward them to the appropriate application and wrap the replies back in the multiplexing protocol.
In summary: please describe the problem your are trying to solve, not only one small technical detail of it.
The problem is that I can't bind to the same network port with two programs at once
Yes you can. Just set SO_REUSEADDR and maybe SO_REUSEPORT on both of them before you bind.
I am programming a server and client program to communicate between a windows PC using the Boost libraries and a Linux ARM beagleboard using the asio stand alone libraries. I have for a while had successful UDP communication between the two devices but now I want to recover the port from the endpoint the server discovers when the client connects. The way the client connects is via query:
udp::resolver resolver(io_service);
udp::resolver::query query_tx(udp::v4(), hostIP, "43210");
udp::endpoint receiver_endpoint_tx = *resolver.resolve(query_tx);
where host IP is a string and this works fine. Upon debugging though I notice that when i check the value returned by:
receiver_endpoint_tx.port()
This returns 51880. Now don't jump the guns and yell out network byte order and host byte order. I AM AWARE. The strange part is that this number 51880 sometimes is a different number and when i check what the server has stored in its endpoint it is a completely different number: 21743. Now I know I must be doing something wrong with the byte orders but i tried:
//unsigned long port_long = boost::asio::detail::socket_ops::host_to_network_long(receiver_endpoint_tx.port());
//unsigned long port_short = boost::asio::detail::socket_ops::host_to_network_short(receiver_endpoint_tx.port());
And they do not give me back my original port: 43210. Neither does network to host. So what am i missing and how can I on both ends recover my 43210 port? Obviously it must be there somewhere because they are successfully communicating.
Thanks in advance, sorry if noob mistake :)
Fistly, UDP is connectionless, there is no connection.
I'm not sure if I understand you correctly, but it sound too me like you want to bind to specific port numbers. If you want the client to send a packet from port x to port y on the server, and the server should respond from port y to port x, then you need to bind the sockets to the desired ports. Alternatively you can use the constructor to bind. Not doing so will result in the OS using ephemeral ports.
Further, to get the remote endpoint that a packet was received from the async_receive_from takes the sender_endpoint reference parameter. When the read handler is called, you can retrieve host and port from it.