DPDK MLX5 PMD driver probe issue - networking

Question

I'm not able to use the mlx5 pmd driver with some Mellanox NICs I have installed on my server. The error I'm receiving during EAL initialization is:
et_mlx5: no Verbs device matches PCI device 0000:03:00.0, are kernel drivers loaded?
The DPDK version I'm currently using is: DPDK-STABLE-18.11
I have installed the OFED latest version:
mlnx-en-4.5-1.0.1.0-ubuntu16.04-x86_64
I have performed modprobe of the ib_uverbs kernel module
Here's the kernel version I'm using
moragalu#server:~$ uname -r
4.4.0-143-generic
Here are the NIC models:
moragalu#server:~$ lspci | grep Mell
03:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
03:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
06:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
06:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
The firmware version that the NICs are using:
moragalu#eridium03:~$ ethtool -i eridium25-03
driver: mlx5_core
version: 4.5-1.0.1
firmware-version: 14.24.1000 (MT_2420110034)
expansion-rom-version:
bus-info: 0000:06:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
The complete output is of the eal initialization is:
EAL: Detected 16 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL: probe driver: 15b3:1015 net_mlx5
net_mlx5: no Verbs device matches PCI device 0000:03:00.0, are kernel drivers loaded?
EAL: Requested device 0000:03:00.0 cannot be used
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL: probe driver: 15b3:1015 net_mlx5
net_mlx5: no Verbs device matches PCI device 0000:03:00.1, are kernel drivers loaded?
EAL: Requested device 0000:03:00.1 cannot be used
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL: probe driver: 15b3:1015 net_mlx5
net_mlx5: no Verbs device matches PCI device 0000:06:00.0, are kernel drivers loaded?
EAL: Requested device 0000:06:00.0 cannot be used
EAL: PCI device 0000:06:00.1 on NUMA socket 0
EAL: probe driver: 15b3:1015 net_mlx5
net_mlx5: no Verbs device matches PCI device 0000:06:00.1, are kernel drivers loaded?
EAL: Requested device 0000:06:00.1 cannot be used
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL: probe driver: 15b3:1015 net_mlx5
net_mlx5: no Verbs device matches PCI device 0000:03:00.1, are kernel drivers loaded?
EAL: Driver cannot attach the device (03:00.1)
EAL: Failed to attach device on primary process
Current modules loaded in the kernel:
moragalu#eridium03:~$ lsmod | grep ib
mlx5_ib 16384 0
mlx_compat 24576 4 mlx4_en,mlx5_ib,mlx4_core,mlx5_core
ib_uverbs 61440 0
ib_iser 49152 0
rdma_cm 49152 1 ib_iser
ib_cm 49152 1 rdma_cm
ib_sa 36864 2 rdma_cm,ib_cm
ib_mad 49152 2 ib_cm,ib_sa
ib_core 106496 7 rdma_cm,ib_cm,ib_sa,iw_cm,ib_mad,ib_iser,ib_uverbs
ib_addr 20480 2 rdma_cm,ib_core

Answer 1

I've had the same problems when using the RDMA-core libraries for the ibverbs dependency. In the past I've managed to find a bug in mlx5_core.c (hardcoded the no of queues to 8 in the probe function and it magically worked), but I'm not sure it's the same issue for you.
Either way, the problem went away when I installed the latest Mellanox OFED Drivers, so it's a good idea to try that. Just remember to install it using the command:
mlnxofedinstall --dpdk --upstream-libs
edit: Just noticed you have the drivers installed - make sure you did the installation as above. One more thing you can do: check the output of this (compiled with -libverbs):
#include <infiniband/verbs.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>
int main() {
struct ibv_device ** devices;
int num;
struct ibv_context * ctx;
devices = ibv_get_device_list(&num);
int i;
if(devices[0] == NULL)
printf("devices is null\n");
printf("got %d devices\n", num);
for (i=0;i<num;i++) {
printf(ibv_get_device_name(devices[i]));
printf("\n");
ctx = ibv_open_device(devices[i]);
if (ctx == NULL)
printf("ctx is null \n");
else
printf("device opened\n");
}
if (errno)
printf("ERROR: %s\n", strerror(errno));
ibv_free_device_list(devices);
return 0;
}
If it lists no devices at least you'll know it's an issue with the verbs drivers and not DPDK itself.