I'm trying to find out if there is any native TCP/IP stack implemetation provided in DPDK or any popular open-source project to achieve it. Any help will be grateful.
Update:
My platform is Ubuntu 16.04 x64, Intel 82599es NIC with DPDK 20.08. What I'm trying to accomplish is to rebuild TCP connections out of the packets I receive on the NIC port for later use. I thought tools like ANS, mTCP and fstack may do the track but they are third party and some of them are not fully open-sourced, so I'm looking for a native one or one that is popular for developers. I don't know what should I call this kind of requirement, sorry for troubles causing because of the question, I'll change it after I know the related concept better.
There are no native TCP-IP stack implementation in DPDK version till date DPDK version 20.11 LTS. Going further in my humble opinion DPDK would not be implementing native TCP-IP stack. Hence current options are
Since the requirement is to rebuild TCP connection state information, my recommendation is to
create RTE_FLOW_ACTION_MIRROR to create the packet copy at HW NIC level for all interested TCP connection using a combination of IP-TCP address-ports.
If there is no HW option, either using ref_cnt_update or copying the user packet payload create a copy of the packet.
With help RTE_RINGS or RTE_FB_ARRAY organize the packets from client and server based on symmetric RSS (if available) or based on custom HASH to appropriate containers.
For you packet processing recommendation is either use FSTACK or mTCP or BSD TCP-ip from scratch
there are multiple references on the Internet which gives hints to get started too. Please refer
mTCP slide 14
fstack slide deck
Related
I'm trying to learn network protocol stack(ie. Transport, IP, datalink layer library code implementation) along with linux. I'm confused where to start.
First question is whether these codes come as in-built features of linux kernel/above library layers.
If so why I can see 3rd party protocol stack in some applications (by blunk micro system - developer of protocol stack)
If Linux doesn't have it as core feature, is linux give only placeholders for network part(like just Macros to enable the 3rd party stack ). But an article says it has Net4 networking codebase.
If linux has in-built network features what are the linux modules I need to go through or where to start? Not only in the network perspective, if i'm guided to explore in linux in all aspects (process, memory, drivers) in the "code level", it would be helpful please.
Note: I'm greedy to write my own OS and protocol stack hence trying to understand an existing system.
Thanks in advance!
First question is whether these codes come as in-built features of linux kernel/above library layers.
Linux kernel has network stack up to including layer 4, i.e., TCP and UDP (well, kernel + a set of utilities needed to configure it). I think DNS is in kernel too, but I am not so sure. TLS used to be implemended as a library (OpenSSL and GnuTLS are I think the most common ones), but there seems to be kernel part too now (link.
Note, that some of the TCP functionality is offloaded to the network card (hardware). At high speeds (1Gb+) you won't get full performance without these features.
I am not familiar with all VoIP related protocols, but I think they are libraries, not kernel.
If so why I can see 3rd party protocol stack in some applications (by blunk micro system - developer of protocol stack)
I believe the reason is performance. If you implement a custom stack with a subset of features, it might work better for your applications. Also there are advanced features and protocols that might not be available in the kernel itself.
If Linux doesn't have it as core feature, is linux give only placeholders for network part(like just Macros to enable the 3rd party stack ). But an article says it has Net4 networking codebase.
there is a very large codebase
If linux has in-built network features what are the linux modules I need to go through or where to start? Not only in the network perspective, if i'm guided to explore in linux in all aspects (process, memory, drivers) in the "code level", it would be helpful please.
hmmm, this is a very good question, and I don't think there is an easy answer. In my experience reading the code is the only way to figure this out. However some people tried to fish LWN.net for information.
you could probably start somewhere here: include/net/
First question is whether these codes come as in-built features of linux kernel/above library layers.
If linux has in-built network features what are the linux modules I need to go through or where to start?
You can think of a protocol stack as of a library. Linux kernel has one which runs inside the kernel address space and uses kernel APIs unavailable in user-space: https://github.com/torvalds/linux/tree/master/net/ipv4
There are multiple in-depth books about Linux kernel networking. Reading one is required for good understanding.
If so why I can see 3rd party protocol stack in some applications (by blunk micro system - developer of protocol stack)
Zero-copy, low-latency and streaming (processing an Ethernet packet in CPU-L1-cache-line-sized chunks while it hasn't been read off the wire in full) networking have been problematic with Linux kernel network stack. For these reasons makers of networking hardware offered their own user-space network stacks, aka kernel bypass.
Linux kernel network stack is getting better these days with MSG_ZEROCOPY and io_uring.
I'm currently looking at options to allow me to build a remote COM-port solution.
The idea is to be able to access from my remote PC, another PC that's directly connected to a device locally via its serial COM-port.
I know that the obivous answer is to use a VPN between the 2 Internet connected PCs.
However, I need this solution to be as seamless to the end-user as possible.
i.e. no installing and configuring VPN software, etc.
So I was thinking that WebRTC would be great because the end-user can simply use their web-browser and not have to install any additional software.
My question is, is it possible to stream the COM port data between the 2 PCs via WebRTC?
If so, can you please point me in the right direction as to how I can go about achieving this?
Sorry if this is a ridiculous question, I'm very new to WebRTC, just exploring my options.
Thanks.
That should work great!
Networking wise you get NAT Traversal. That means the two computers can be in completely different networks, and still communicate. You may have to run a TURN server if P2P isn't possible.
Data wise you can exchange anything you want via data channels. It is datagram based and you can send/receive binary data. You get a callback telling you how much has been delivered, that way you can detect backpressure.
Are you ok with installing software on the remote host? You can do something like Pion WebRTC's data-channels. This shows you can have a browser connect to a Go process via WebRTC. Then use tarm/serial on the remote host to interact with the device.
If you want a browser on both ends there is the Web Serial API I haven't used it myself though. That locks you into only doing Chromium which might be an issue.
I've been asked to implement VPN capabilities in an existing software project on an embedded system, in order to make the device available via network to an external server while avoiding trouble with firewalls (no need for encryption, just to make it accessible).
Unfortunately, the embedded system is based on a Cortex-M4 MCU, therefore Linux, which would allow for VPN nearly out of the box, is not an option. All I've got is an RTOS and a working LwIP stack.
I've used VPNs in the past. However, my network knowledge is rather limited concerning implementing VPNs, so I'm rather stumped. As I think, I'd use the current LwIP instance for building up the tunnel connection, and the application would use a second instance for the actual network communication, while the network interface of the second instance is a virtual one (like a tap device on linux), encapsulating its low level data and tranceiving it via the tunnel connection of the first LwIP instance.
Maybe this way I'd be able to create a custom solution for the problem, but the solution should conform to any standards (as the server will be any kind of sophisticated system).
So I wonder if anyone has been confronted with a task like this, and would appreciate any hint what to do, at least a direction where to look at.
Thanks in advance!
Are there any libraries which put a reliability layer on top of UDP broadcast?
I need to broadcast large amounts of data to a large number of machines as quickly as possible, and generally it seems like such a problem must have already been solved many times over, but I wasn't able to find anything except for the Spread toolkit, which has a somewhat viral license (you have to mention it in all materials advertising the end product, which I'm not sure our customer will be willing to do).
I was already going to write such a thing myself (because it would be extremely fun to do!) but decided to ask first.
I looked also at UDT (http://udt.sourceforge.net) but it does not seem to provide a broadcast operation.
PS I'm looking at something as lightweight as a library - no infrastructure changes.
How about UDP multicast? Have a look at the PGM protocol for which there are several commercial and open source implementations.
Disclaimer: I'm the author of OpenPGM, an open source implementation of said protocol.
Though some research has been done on reliable UDP multicasting, I haven't yet used anything like that. You should take into consideration that this might not be as trivial as it first sounds.
If you don't have a list of nodes in the target network you have no idea when and to whom to resend, even if active nodes receiving your messages can acknowledge them. Sending to a large number of nodes, expecting acks from all of them might also cause congestion problems in the network.
I'd suggest to rethink the network architecture of your application, e.g. using some kind of centralized solution, where you submit updates to a server, and it sends this message to all connected clients. Or, if the original sender node's address is known a priori, then just let clients connect to it, and let the sender push updates via these connections.
Have a look around the IETF site for RFCs on Reliable Multicast. There is an entire working group on this. Several protocols have been developed for different purposes. Also have a look around Oracle/Sun for the Java Reliable Multicast Service project (JRMS). It was a research project of Sun, never supported, but it did contain Java bindings for the TRAM and LRMS protocols.
I'm trying to create an application to craft packets to be able to debug some gateways here, and to experiment with TCP DoS situations.
Nevertheless this should be very easy, I didn't find a way to implement this for a Windows application.
I started using Impacket from Core Security in Python on a Unix box, but I want to avoid this for now. First of all Impacket doesn't work for Windows, and it doesn't seem to do exactly what I want.
Does anyone know how to get a simple raw-socket like behavior in Windows? I know that there're no Raw sockets any more. But is there something similar? Any C# library I can use... I didn't find anything jet.
Thanks ;)
There's not a lot to creating the socket.
using System.Net.Sockets;
Socket rawSocket = new Socket(AddressFamily.InterNetwork, SocketType.Raw, ProtocolType.Raw);
or if it's custom TCP packets you're after:
Socket rawSocket = new Socket(AddressFamily.InterNetwork, SocketType.Raw, ProtocolType.IP);
If you're planning on sending IP or higher layer packets that's not exposed by the .Net framework. However IP and TCP packets are pretty simple to put together and if you're testing malformed packets you'll most likely need to customise the packets anyway.
Try to use libpcap (winpcap), it can work under the tcp/ip stack, just on raw packet level.