I wanted to try-out multiple RX/TX queue in KVM (Guest: CentOS). I have compiled DPDK (version: 18.05.1) and inserted igb_uio driver (bound two interface to it).
I am trying client to server connection (private).
client (eth1: 10.10.10.2/24) <--> (eth1) CentOS VM (DPDK: 18.05.1)
(eth2) <--> server (eth1: 10.10.10.10/24)
VM manages both interface directly in passthrough mode (macvtap - passthrough).
<interface type='direct' trustGuestRxFilters='yes'>
<source dev='ens1' mode='passthrough'/>
<model type='virtio'/>
<driver name='vhost' queues='2'/>
</interface>
When l2fwd applications started with single RX & TX queue (default change) & no-mac-updating. Client & server connectivity works perfectly.
I made some changes to try multiple RX/TX queues with l2fwd application.
I could see that ARP is not getting resolved at either end. VM doesn't receive any packets afterwards.
Can someone point me to the document to use multiple RX/TX queues which can verify my changes? Does multiple RX/TX queues work in VM environment? I have seen others also complaining about it.
I am newbie in DPDK world. Any help will be useful. Thank you.
Edited (Adding more details):
I am configuring ethernet device with 1 RX queue and 2 TX queues in l2fwd example.
uint16_t q = 0;
uint16_t no_of_tx_queues = 2;
// Configuring Ethernet Device
rte_eth_dev_configure(portid, 1, no_of_tx_queues, &local_port_conf);
// Configuring Rx Queue
ret = rte_eth_rx_queue_setup(portid, 0, nb_rxd, rte_eth_dev_socket_id(portid), &rxq_conf, l2fwd_pktmbuf_pool);
// Configuring 2 TX Queue
for(q = 0; q < no_of_tx_queues; q++) {
ret = rte_eth_tx_queue_setup(portid, q, nb_txd, rte_eth_dev_socket_id(portid), &txq_conf);
}
I am reading packets from single RX queue: Queue-id: 0 (as setup earlier).
nb_rx = rte_eth_rx_burst(portid, 0, pkts_burst, MAX_PKT_BURST);
I am seeing that some packets are coming and forwarded to other interface but some are not. For ICMP (ping), I can see ARP is forwarded but ICMP echo request is not read by l2fwd.
Solution what I found:
I have configured 2 RX & 2 TX queues in l2fwd. I can ICMP request is read from second RX queue (Queue-id: 1) and forwarded too. With that, client to server connectivity is working as expected.
The question here is:
Even I have configured 1 RX queue & 2 TX queue. Why few packets are coming on Queue-id: 1 (which is not configured & not read by l2fwd application).
It is observed in KVM (running on CentOS) environment. I have checked the same on ESXI, I can see all packets are read from single queue (Queue-id: 0) and forwarded.
Why?? Please explain. Is there any way I can turn off load balancing (of packet transmitted on two RX queues) in KVM so that I can receive all the packets on single queue?
Here is the DPDK's Vhost multiple queues test plan with all the command line arguments used:
https://doc.dpdk.org/dts/test_plans/vhost_multi_queue_qemu_test_plan.html
There are no much details in the question, so the only suggestion I have is to make sure multiple queues work first and then run l2fwd on top of that. If the guest OS does not work with multiple queues, DPDK won't fix the issue.
Related
I have a question for you. I know that vhost-user NICs can be configured with many RX/TX queues, but does the vhost-user driver ensures the distribution of traffic between RX queues?
I used the sample application l3fdw to switch traffic between two vhost-user NICs, each with 4 queues. The traffic was generated using TREX (and testpmd also), running inside a VM. When I traced my experiment, I noticed that the traffic was only received in queue "0", while the other RX queues were empty.
The l3fdw app tells me that "Port 0 modified RSS hash function based on hardware support,requested:0xa38c configured:0". For offloading capabilities, testpmd indicates that the vhost-user nic NIC has only support for VLAN STRIP (and not for RSS)!
I appreciate any clarification on this matter.
Thank you,
PS:
DPDK version: 19.08
Qemu version: 4.2.1
Adele
Answer for the original question does the vhost-user driver ensures the distribution of traffic between RX queues? is
There is no mechanism like RSS or RTE_FLOW from DPDK Libraries which will ensure software packet distribution on the RX queues of VHOST NIC.
#AdelBelkhiri there are multiple aspects to be clarified to understand this better.
Features supported by VHOST PMD do not advertise either RTE_FLOW or RSS.
Driver code for vhost pmd in file rte_eth_vhost.c does not advertise RSS or RTE_FLOW capability
there is an article which describes the use of OVS and Multiple queues. The RSS is configured on the Physical NIC with 2 RX queues. The RSS is done on the Physical NIC, 2 separate threads picks the packets from the Physical RX queue and puts the same in VHOST queues. Thus achieving pass-through RSS.
hence in your case where you have 2 VM with 2 NIC ports each having 4 queues, please try 8 PMD threads on OVS to concurrently fwd packets between queues. Where the TREX (TX) VM will ensure to put appropriate packets into each queue seperately.
But the simple answer is there is no RSS or RTE_FLOW logic to distribute traffic
Is it possible to build a modbus-RTU ethernet gateway using a linux box or arduino?
I have plc slaves linked together using modbus RTU, and I want to connect one of them over TCP with my pc as master.
I wonder if I can use a linux box (rasbery pi/raspbian) connected to a router, as a modbus-to-tcp converter by piping the usb port to my local ip on some port, as one plc will be connected in modbus rtu to the linux box`s usb port.
Piping command will be something Like this:
nc -l 5626 > /dev/ttyUSB0 < /dev/ttyUSB0
My goal is to connect a pc (networked with the linux box) to that plc through the linux box using modbus.
Yes you can use a linux box with ethernet port.
Its not a simple 'nc' is it.
First things first, you need to know, Does your PLC's support Modbus TCP or Modbus RTU over TCP.
bcos both the formats are not the same, they are not interchangeable.
once you ascertain this, you need to write a TCP Client to Connect to the Slaves as they run the Servers.
If they don't support Modbus TCP yet, you need to write a TCP servers as well.
on top of that, you have the handshaking delays and half open connections and what not.
If it is a commercial device, you need to look at the setup it will be running as well.
Hope this helps
I'm not entirely sure as to what your requirements are, but you may want to have a look at the following library. It's a Modbus TCP->RTU library I assume you can use.
https://github.com/3cky/mbusd
Best Regards
While it's certainly possible to build a Modbus TCP/RTU gateway, you won't be able to do it with a simple nc command, for two reasons:
The actual Modbus protocol data unit (PDU), merely containing the Modbus function code and data, is embedded in an application data unit (ADU) so the request reaches the correct device free of communication errors. This ADU is different for RTU and TCP. Generically, the ADU looks like this:
------------------------------------------
| Additional address | PDU | Error check |
------------------------------------------
For RTU, the "Additional address" is a single byte identifying the unit/slave on the serial line, and the "Error check" is a 16-bit CRC.
For TCP, the "Additional address" is the 7-byte Modbus application protocol header (MBAP), consisting of transaction identifier, protocol identifier, and a length field (2 bytes each), plus a single byte identifying the unit/slave (usually 255, though for a gateway, this would be the ID of the RTU slave behind it). The "Error check" is empty (zero bytes) as that is already taken care of by TCP.
The communication semantics are different for RTU and TCP.
For RTU, you write a request to the serial line, wait for the reply, and only then write the next request. There must be 3½ characters of silence between frames.
For TCP, you can in principle have multiple connections being served concurrently. If you tried to forward two TCP requests to the serial line simultaneously, chaos would ensue.
Still, you can build a working gateway with a slightly more complicated program than nc:
Accept connections on TCP port 5626 (note that the actually recommended port number for Modbus TCP is 502), convert the received TCP ADUs to RTU ADUs and put them into a queue together with a back channel.
Another part of your program takes one item at a time from that queue, sends the ADU over the serial line and reports the result back through the back channel. Then it goes on the to the next item, and so on.
The results arriving on the back channels are converted to TCP ADUs and sent back on the respective TCP connection.
This would certainly work on a Raspberry Pi, and possibly also on Arduino, depending on how large you want your queue to be.
Sources:
http://www.modbus.org/docs/Modbus_Application_Protocol_V1_1b.pdf
http://www.modbus.org/docs/Modbus_Messaging_Implementation_Guide_V1_0b.pdf
I have CentOS on my server and the server has two NIC.
I want to capture any packets on NIC number 1 and forward them to NIC number 2. I use Socket in my C++ application to capture and froward the packets, then I use IPerf to analyze my application. so I connect each of the NICs to different computers (Computer A and B) and try to send UDP packets from computer A to B. The server suppose to capture packets from A then forward them to B and vice versa.
It works well when I'm pinging computer B from A, but there are some things wrong when I try to generate more packets via IPerf. IPerf generate 100K UDP packets per second (22 bytes payload) and send it from A to B, but computer B does not receive all of the packets in a same time! for example if IPerf on A send 100K packets at the first second, Computer B receive these packets at 20 seconds! It seems like some caching system on server that holds the received packet! let me just show you what happened:
[OS: CentOS]
[00:00] Computer A Start Sending...
[00:01] Computer A send 100,000 packets
[00:01] Finnish
[00:00] Computer B start to listen...
[00:01] Computer B receive 300 packets
[00:02] Computer B receive 250 packets
[00:03] Computer B receive 200 packets
[00:04] Computer B receive 190 packets
[00:05] Computer B receive 180 packets
[00:06] Computer B receive 170 packets
[00:07] Computer B receive 180 packets
[00:08] Computer B receive 20 packets
[00:09] Computer B receive 3 packets
[00:10] (same things happened until all 100K packets will receive, it takes long time)
At the 4th second, I unplugged the network cable from computer A to make sure it's not sending any packets any more, after that computer B is still receiving packets! It seems like something holds the traffic at the server and releases it slowly. I tried to turn the firewall off but nothing has been changed.
I changed the server OS and use Ubuntu to check if there is any problem in my code, but it works well on Ubuntu. after that I tried to change CentOS sockets buffer size but it doesn't help. here is some of important part of my code:
How I setup a socket:
int packet_socket = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL));
if (packet_socket == -1) {
printf("Can't create AF_PACKET socket\n");
return -1;
}
// We would use V3 because it could read/pool in per block basis instead per packet
int version = TPACKET_V3;
int setsockopt_packet_version = setsockopt(packet_socket, SOL_PACKET, PACKET_VERSION, &version, sizeof(version));
if (setsockopt_packet_version < 0) {
printf("Can't set packet v3 version\n");
return -1;
}
//Set Interface in Promiscuous Mode
struct ifreq ifopts;
strncpy(ifopts.ifr_name,interface_name,IFNAMSIZ-1);
ioctl(packet_socket,SIOCGIFFLAGS,&ifopts);
ifopts.ifr_flags|=IFF_PROMISC;
ioctl(packet_socket,SIOCGIFFLAGS,&ifopts);
//Bind Socket On Specific Interface
struct sockaddr_ll sockll;
bzero((char *)&sockll,sizeof(sockll));
sockll.sll_family=AF_PACKET;
sockll.sll_protocol=htons(ETH_P_ALL);
sockll.sll_ifindex=get_interface_index(interface_name,packet_socket);
bind(packet_socket,(struct sockaddr *) &sockll , sizeof(sockll));
And here I receive packets:
u_char *packet;
packet=new u_char[1600];
*length = recv(packet_socket ,packet,1600,0);
And here I send packets:
int res = write(packet_socket ,packet,PacketSize);
I'm so confused and I don't know what is going on CentOS. Can you please help me to understand what is happening?
Is CentOS a good choice for doing this job?
Thank you.
Try disabling selinux on centos
setenforce 0
Then try
I have installed npcap driver which supports loopback interface. I installed it because I need to inyect packets to loopback interface and read them from it. I can easily read packets in the loopback with "pcap_next_ex" as I can do in winpcap in ethernet interfaces, but when I want to inyect a packet to the loopback with "pcap_sendpacket" dosent work, and the function returns 0 (successfull).
I verified this by opening wireshark and watching the packets incoming to the interface, when I use pcap_sendpacket on ethernet interface I can watch the packets, but in the loopback they dont appear when I tried to inject them. Why?
//The array which contains the data of the test package
UCHAR packet[] = "\x..";
//loopback adapter is already opened here
for (int i = 0; i < 100; i++)
printf("%d ", pcap_sendpacket(loopbackAdapter, packet, sizeof(packet)));
I use this code on loopback interface and didnt work (but pcap_sendpacket returned always success), because in wireshark the packets didnt appear, but in ethernet interfaces the injection was successfull.
Does npcap support loopback packet inyection?
Thank you and regards!.
You need to understand that every network packet (even a loopback one) has a direction: on send path (Tx) or on receive path (Rx). Usually, when you send a packet to Npcap Loopback Adapter, you are actually sending it to the Tx path of the Windows TCP/IP stack. When you are sending Tx packets, you usually wait for a localhost application (or a protocol, a driver, etc) responding to them.
I don't know why you want to inject packets to loopback interface and read them from it. Npcap's low-level logic for this part is just not letting the current session (a pcap_t) receive the packets that this session injects. But other sessions can see them, this is why Wireshark can see the packets that you inject.
I just don't know why you are doing this. You seem not to want any other applications respond to these packets. But as a workaround, I think you can get what you want by injecting the packets to the Rx path. By sending packets to Rx, it means to fool Windows to believe that these packets are received from outside. I don't remember how I implement it, but the same session should be able to see these packets in Rx.
We didn't document the Rx feature very clearly. There's only some description in releases: v0.05-r6 and v0.05-r7, and an example here: https://github.com/hsluoyz/UserBridge. They should work even in latest Npcap.
I'm currently encountering an issue in RedHat 6.4. I have two physical NICs which I am trying to bond together using LACP.
I have the corresponding configuration set up on my switch, and I have implemented the recommended configuration from the RedHat Install Guide on my NICs.
However, when I start my network services, I'm seeing my LACP IP on the physical NICs as well as the bonding interface (respectively eth0, eth1 and bond0). i'm thinking I should only see my IP address on my bond0 interface?
The connectivity with my network is not established. I don't know what is wrong with my configuration.
Here are my ifcfg-eth0, eth1 and bond0 files (IP blanked for discretion purposes).
ifcfg-eth0 :
DEVICE=eth0
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
USERCTL=no
TYPE=Ethernet
NM_CONTROLLED=no
ifcfg-eth1 :
DEVICE=eth1
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
USERCTL=no
TYPE=Ethernet
NM_CONTROLLED=no
ifcfg-bond0 :
DEVICE=bond0
IPADDR=X.X.X.X
NETMASK=255.255.255.0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
NM_CONTROLLED=no
BONDING_OPTS="mode=4"
Thanks to anyone who can pinpoint my problem.
Jeremy
Let me answer my own question here in case anyone is having the same issue.
Turns out I just needed to deactivate the "NetworkManager" service on my RedHat Server. Turn it off, and deactivate it, then works like a charm.
Network bonding : Modes of bonding
Modes 0, 1, and 2 are by far the most commonly used among them.
Mode 0 (balance-rr)
This mode transmits packets in a sequential order from the first available slave through the last. If two real interfaces are slaves in the bond and two packets arrive destined out of the bonded interface the first will be transmitted on the first slave and the second frame will be transmitted on the second slave. The third packet will be sent on the first and so on. This provides load balancing and fault tolerance.
Mode 1 (active-backup)
This mode places one of the interfaces into a backup state and will only make it active if the link is lost by the active interface. Only one slave in the bond is active at an instance of time. A different slave becomes active only when the active slave fails. This mode provides fault tolerance.
Mode 2 (balance-xor)
Transmits based on XOR formula. (Source MAC address is XOR’d with destination MAC address) modula slave count. This selects the same slave for each destination MAC address and provides load balancing and fault tolerance.
Mode 3 (broadcast)
This mode transmits everything on all slave interfaces. This mode is least used (only for specific purpose) and provides only fault tolerance.
Mode 4 (802.3ad)
This mode is known as Dynamic Link Aggregation mode. It creates aggregation groups that share the same speed and duplex settings. This mode requires a switch that supports IEEE 802.3ad Dynamic link.
Mode 5 (balance-tlb)
This is called as Adaptive transmit load balancing. The outgoing traffic is distributed according to the current load and queue on each slave interface. Incoming traffic is received by the current slave.
Mode 6 (balance-alb)
This is Adaptive load balancing mode. This includes balance-tlb + receive load balancing (rlb) for IPV4 traffic. The receive load balancing is achieved by ARP negotiation. The bonding driver intercepts the ARP Replies sent by the server on their way out and overwrites the src hw address with the unique hw address of one of the slaves in the bond such that different clients use different hw addresses for the server.
~]#service NetworkManager stop | chkconfig NetworkManager off
Try This and if not continue with bellow command too
~]#service network start/restart | chkconfig network on