I'm learning network communications and already familiar with TCP/IP networking layers (physical, data link ... and application layers) and how data moves through this nodes. But I have some questions about what happens inside a machine, when data is received by a Network Interface Card(NIC).
Questions:
How CPU knows that data from other machine is arrived?
How CPU informs OS that data from other machine is arrived?
How OS knows which application the data is for?
Please, give me some deep explanation for this topic, or advice some useful materials to make it clear.
To give you a general view from Linux point(should be similar for other OS):
The packets arrive in NIC. These packets are copied into circular queues in RAM via DMA. The arrival of packets will generate an interrupt to let the system know that their are packets in RAM. Corresponding to the interrupt there will be an interrupt handler routine registered with the Operating System via the network driver. (To keep things simple didn't talk about softirq's). Each CPU has a poll function whose job is to harvest packets from these queue's and pass them onto upper n/w layers. So answering your queries:
How CPU knows that data from other machine is arrived?
When interrupt occurs and poll loop is not running on the CPU, the OS(via network driver)
will ask the CPU to start the poll loop for harvesting the packets.
How CPU informs OS that data from other machine is arrived?
CPU doesn't need to inform OS. The OS will know when the interrupt occurs as the interrupt handler is a part of the network driver which is part of OS. Infact in a way OS will tell the CPU to start harvesting packets.
How OS knows which application the data is for?
The communication is done via sockets which will have a port number. The packets arrived will have a port number which will guide the OS to take the packet to the required application.
Related
I have a quite newbie question : assume that I have two devices communication via Ethernet (TCP/IP) at 100Mbps. In one side, I will be feeding the device with data to transmit. At the other side, I will be consuming the received data. I have the ability to choose the adequate buffer size of both devices.
And now my question is : If data consumption rate from the second device, is slower than data feeding rate at the first one, what will happen then?
I found some, talking about overrun counter.
Is there anything in the ethernet communication indicating that a device is momently busy and can't receive new packets? so I can pause the transmission from the receiver device.
Can some one provide me with a document or documents that explain this issue in detail because I didn't find any.
Thank you by advance
Ethernet protocol runs on MAC controller chip. MAC has two separate RX-ring (for ingress packets) and TX-ring(for egress packets), this means its a full-duplex in nature. RX/TX-rings also have on-chip FIFO but the rings hold PDUs in host memory buffers. I have covered little bit of functionality in one of the related post
Now, congestion can happen but again RX and TX are two different paths and will be due to following conditions
Queue/de-queue of rx-buffers/tx-buffers is NOT fast compared to line rate. This happens when CPU is busy and not honer the interrupts fast enough.
Host memory is slower (ex: DRAM and not SRAM), or not enough memory(due to memory leak)
Intermediate processing of the buffers taking too long.
Now, about the peer device: Back-pressure can be taken care in the a standalone system and when that happens, we usually tail drop the packets. This is agnostics to the peer device, if peer device is slow its that device's problem.
Definition of overrun is: Number of times the receiver hardware was unable to handle received data to a hardware buffer because the input rate exceeded the receiver’s ability to handle the data.
I recommend pick any MAC controller's data-sheet (ex: Intel's ethernet Controller) and you will get all your questions covered. Or if you get to see device-driver for any MAC controller.
TCP/IP is upper layer stack sits inside kernel(this can be in user plane as well), whereas ARPA protocol (ethernet) is inside MAC controller hardware. If you understand this you will understand the difference between router and switches (where there is no TCP/IP stack).
I have a solarflare nic with paired rx and tx queues (8 sets, 8 core machine real machine, not hyperthreading, running ubuntu) and each set shares an IRQ number. I have used smp_affinity to set which irqs are processed by which core. Does this ensure that the transmit (tx) interrupts are also handled by the same core. How will this work with xps?
For instance, lets say the irq# is 115, set to core 2 (via smp_affinity). Say the nic chooses tx-2 for outgoing tcp packets, which also happens to have 115 irq number. If I have an xps setting saying tx-2 should be accessible by cpu 4, then which one takes precedence - xps or smp_affinity?
Also is there a way to see/set which tx queue is being used for a particular app/tcp connection? I have an app that receives udp data, processes it and sends tcp packets, in a very latency sensitive environment. I wish to handle the tx interrupts on the outgoing on the same cpu (or one on the same numa node) as the app creating this traffic, however, I have no idea how to find which tx queue is being used by this app for this purpose. While the receive side has indirection tables to set up rules, I do not know if there is a way to set the tx-queue selection and therefore pin it to a set of dedicated cpus.
You can tell the application the preferred CPU by setting the cpu affinity (taskset) or numa node affinity, and you can also set the IRQ affinities (in /proc/irq/270/node, or by using the old intel script floating around 'set_irq_affinity.sh' which is on github). This won't completely guarantee which irq / cpu is being used, but it will give you a good head start on it. If all that fails, to improve latency you might want to enable packet steering in the rxqueue so you get the packets in quicker to the correct cpu (/sys/class/net//queues/rx-#/rps_cpus and tx-#/xps-cpus). There is also the irqbalance program and more....it is a broad subject and i am just learning much of it myself.
I am developing a logic core to perform data transfer between a FPGA and a PC over ethernet, using a LAN8710 PHY on my FPGA board.
I've achieved to transfer some UDP data packets from the FPGA to the PC. It's a simple core that complies with the PHY transfer requirements. It builds the UDP package and transfer it to the PC.
To check the reception on the PC, I am using Wireshark and as said above, I receive the packets correctly. I've checked the reception with a simple UDP receiver written by myself.
But, I've noticed that I only receive these packets when Wireshark is running on the PC. I mean, if Wireshark is ON, my application receives the packets too, and the counter of received packets of the following picture increases. (This picture is not mine, just one from the internet)
http://i.stack.imgur.com/wsChT.gif
If I close Wireshark, the PC stops receiving packets and the counter of received packets stops. My application stops receiving too.
Although novice on networking topics, I suspect that this issue is related to PC-side. Seems like Wireshark is "opening/closing" the ethernet communication channel, or something like that. Does anyone knows about this issue?
To build a functional core to transfer data between a PC and the FPGA, I've developed a core to transfer and receive UDP packets. Next step will be ARP implementation (to let the PC identify my FPGA board, as I understand). What protocols are necessary to perform full-duplex data transfer between this 2 devices?
Thank you very much in advance,
migue.
Check whether you are able to get appropriate receive interrupt at ethernet driver level on PC-side for a single transmitted packet by FPGA. If you do not get the receive interrupt, check on the transmit side(FPGA) for appropriate transmit interrupts for packet that is being transmitted. This should mostly help you in cornering the issue.
As far as i know, wireshark is just a packet analyzer/sniffer. However, if wireshark is suspected, one option could be to try with alternate packet sniffer to rule out if any such scenario is happening.
A handy tool for determining problems in network and also for determining the network statistics shall be netstat. netstat -sp udp shall list down the statistics only for UDP. There are many other parameters that can be used with netstat for diagnosis.
After many months I solved it, I post to help someone stucked in the same point.
Finally I figured out that Wireshark uses a tool to access the network link layer of the computer. This tool allows Wireshark to sniff all incoming and outgoing packets at a specified network device. To do this, the first step is to OPEN the network device, and that's why my program only worked if Wireshark was open.
Regards.
I am working on a smartcard reader project here i will have to read/write data from the smartcard reader.
Also i will have to read/write data from PC application.
There are two serial port on my microcontroller one connected to smartcard reader other to PC.
Smartcard reader <------> Microcontroller <-----> PC
I have ported linux & using /ttys0 & /ttys1 driver for this.
1> My question is if application have to find that some data is available to be read from the port than will i have to always check it with read() system call ?
2> Does ttys0 driver have internal buffer to store received data ? Or data is lost if application do not read data immediately ?
3> Here using seprate threads for rx/tx from each port, is it right approach ?
Please guide me i am new to Embedded linux.
//John
Yes, there's a fair amount of buffering on linux tty's.
You have a few choices for how to interact with them.
you can make them non-blocking, and frequently poll to see if you can read data from them (but this may result in uselessly spinning CPU cycles, slowing other tasks)
you can use select() to yield to the scheduler until one of your devices has data for you to act on
you can use blocking I/O, however since you have multiple ports that may also require multiple threads
TTY programming is similar to socket programming in Linux. So basically you can set the socket to be a asynchronous and receive a signal once data is available. Regarding buffering, yes it's buffered using two flipping buffers. You can check chapter 18 in Linux device drivers 3rd edition regarding TTY implementation in the kernel.
I am trying to send packets from one machine to another using tcpreplay and tcpdump.
If I write a driver for capturing packets directly from the NIC, which path will be followed?
1) N/w packet ----> NIC card ----> app (no role of kernel)
2) N/w packet -----> Kernel -----> NIC card ---> app
Thanks
It's usually in this order:
NIC hardware gets the electrical signal, hardware updates some of its registers and buffers, which are usually mapped into computer physical memory
Hardware activates the IRQ line
Kernel traps into interrupt-handling routine and invokes the driver IRQ handling function
The driver figures out whether this is for RX or TX
For RX the driver sets up DMA from NIC hardware buffers into kernel memory reserved for network buffers
The driver notifies upper-layer kernel network stack that input is available
Network stack input routine figures out the protocol, optionally does filtering, and whether it has an application interested in this input, and if so, buffers packet for application processing, and if a process is blocked waiting on the input the kernel marks it as runnable
At some point kernel scheduler puts that process on a CPU and resumes is, application consumes network input
Then there are deviations from this model, but those are special cases for particular hardware/OS. One vendor that does user-land-direct-to-hardware stuff is Solarflare, there are others.
A driver is the piece of code that interacts directly with the hardware. So it is the first piece of code that will see the packet.
However, a driver runs in kernel space; it is itself part of the kernel. And it will certainly rely on kernel facilities (e.g. memory management) to do its job. So "no role of kernel" is not going to be true.