lkml.org 
[lkml]   [2021]   [Jul]   [14]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    Subject[5.14-rc1] mlx5_core receives no interrupts with maxcpus=8
    Date
    Hi all,
    I'm seeing a strange "no MSI-X interrupt" issue with the Mellanox NIC
    driver on a physical Linux host [1], if I only enable part of the CPUs.

    The physical host has 104 logical processors (2 sockets, and each socket
    has 26 cores with HT enabled). By default, the Mellanox driver works fine
    when Linux boots up.

    If I only use 1, 2, 32, 64, 96 processors by the Linux kernel parameter
    "maxcpus=X" or "nr_cpus=X", everthing still works fine.

    However, if the Linux host OS only uses 4, 8 or 16 processors, the
    mlx5_core driver fails to load as it can not receive interrupt when
    creating EQ (maxcpus=8 or 16), or the driver can load but it reports a
    timeout error when I try to bring the NIC up (maxcpus=4). This issue is
    a 100% repro.

    For example, with "maxcpus=8", I get the below timeout error when trying
    to load mlx5_core:

    # modprobe mlx5_core
    [ 1475.716688] mlx5_core 0000:d8:00.0: firmware version: 16.25.8352
    [ 1475.722742] mlx5_core 0000:d8:00.0: 126.016 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x16 link)
    [ 1475.991398] mlx5_core 0000:d8:00.0: E-Switch: Total vports 2, per vport: max uc(1024) max mc(16384)

    [ 1537.020001] mlx5_core 0000:d8:00.0: mlx5_cmd_eq_recover:245:(pid 1416): Recovered 1 EQEs on cmd_eq
    [ 1537.028969] mlx5_core 0000:d8:00.0: wait_func_handle_exec_timeout:1062:(pid 1416): cmd[0]: CREATE_EQ(0x301) recovered after timeout
    [ 1598.460003] mlx5_core 0000:d8:00.0: mlx5_cmd_eq_recover:245:(pid 1416): Recovered 1 EQEs on cmd_eq
    [ 1598.468978] mlx5_core 0000:d8:00.0: wait_func_handle_exec_timeout:1062:(pid 1416): cmd[0]: CREATE_EQ(0x301) recovered after timeout
    [ 1659.900010] mlx5_core 0000:d8:00.0: mlx5_cmd_eq_recover:245:(pid 1416): Recovered 1 EQEs on cmd_eq
    [ 1659.908987] mlx5_core 0000:d8:00.0: wait_func_handle_exec_timeout:1062:(pid 1416): cmd[0]: CREATE_EQ(0x301) recovered after timeout
    [ 1721.340006] mlx5_core 0000:d8:00.0: mlx5_cmd_eq_recover:245:(pid 1416): Recovered 1 EQEs on cmd_eq
    [ 1721.348989] mlx5_core 0000:d8:00.0: wait_func_handle_exec_timeout:1062:(pid 1416): cmd[0]: CREATE_EQ(0x301) recovered after timeout

    When this happens, the mlx5_core driver is stuck with the below
    call-trace, waiting for some interrupt:

    # ps aux |grep modprobe
    root 1416 0.0 0.0 11024 1472 ttyS0 D+ 08:08 0:00 modprobe mlx5_core
    root 1480 0.0 0.0 6440 736 pts/0 S+ 08:15 0:00 grep --color=auto modprobe

    # cat /proc/1416/stack
    [<0>] cmd_exec+0x8a7/0x9b0 [mlx5_core]
    [<0>] mlx5_cmd_exec+0x24/0x50 [mlx5_core]
    [<0>] create_map_eq+0x2a6/0x380 [mlx5_core]
    [<0>] mlx5_eq_table_create+0x504/0x710 [mlx5_core]
    [<0>] mlx5_load+0x52/0x130 [mlx5_core]
    [<0>] mlx5_init_one+0x1cc/0x250 [mlx5_core]
    [<0>] probe_one+0x1d3/0x2a0 [mlx5_core]
    [<0>] local_pci_probe+0x45/0x80
    [<0>] pci_device_probe+0x10f/0x1c0
    [<0>] really_probe+0x1c1/0x3b0
    [<0>] __driver_probe_device+0x109/0x180
    [<0>] driver_probe_device+0x23/0xa0
    [<0>] __driver_attach+0xbd/0x160
    [<0>] bus_for_each_dev+0x7c/0xc0
    [<0>] driver_attach+0x1e/0x20
    [<0>] bus_add_driver+0x152/0x1f0
    [<0>] driver_register+0x74/0xd0
    [<0>] __pci_register_driver+0x68/0x70
    [<0>] init+0x6b/0x1000 [mlx5_core]
    [<0>] do_one_initcall+0x46/0x1d0
    [<0>] do_init_module+0x62/0x250
    [<0>] load_module+0x2503/0x2730
    [<0>] __do_sys_finit_module+0xbf/0x120
    [<0>] __x64_sys_finit_module+0x1a/0x20
    [<0>] do_syscall_64+0x38/0xc0
    [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xae

    To make the issue even weirder, when the issue happens (e.g. when Linux
    only uses 8 processors), if I manually bring CPU #8~#31 online [2] and
    then bring them offline [3], the Mellanox driver will work fine!

    This is a x86-64 host. Is it possibe that the IOMMU Interrrupt Remapping
    is not proprely set up with maxcpus=4, 8 and 16?

    The above tests were done with the recent Linux v5.14-rc1 kernel. I also
    tried Ubuntu 20.04's kernel "5.4.0-77-generic", and the Mellanox driver
    exhibits exactly the same issue.

    I have Linux/Windows dual-boot on this physical machine, and Windows
    doesn't have the issue when I let it only use 4, 8 and 16 processors.
    So this looks like somehow the issue is specific to Linux.

    Can someone please shed some light on this strange issue? I'm ready
    to provide more logs if needed. Thanks!

    PS, the physical machine has 4 NVMe controllers and 4 Broadcom NICs,
    which are not affected by maxcpus=4, 8, and 16.

    [1] This is the 'lspci' output of the Mellanox NIC:
    d8:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
    Subsystem: Mellanox Technologies MT27800 Family [ConnectX-5]
    Flags: bus master, fast devsel, latency 0, IRQ 33, NUMA node 1
    Memory at f8000000 (64-bit, prefetchable) [size=32M]
    Expansion ROM at fbe00000 [disabled] [size=1M]
    Capabilities: [60] Express Endpoint, MSI 00
    Capabilities: [48] Vital Product Data
    Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
    Capabilities: [c0] Vendor Specific Information: Len=18 <?>
    Capabilities: [40] Power Management version 3
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
    Capabilities: [1c0] Secondary PCI Express
    Kernel driver in use: mlx5_core
    Kernel modules: mlx5_core
    00: b3 15 17 10 46 05 10 00 00 00 00 02 08 00 00 00
    10: 0c 00 00 f8 00 00 00 00 00 00 00 00 00 00 00 00
    20: 00 00 00 00 00 00 00 00 00 00 00 00 b3 15 80 00
    30: 00 00 e0 fb 60 00 00 00 00 00 00 00 ff 01 00 00

    [2] for i in `seq 8 31`; do echo 1 > /sys/devices/system/cpu/cpu$i/online; done
    [3] for i in `seq 8 31`; do echo 0 > /sys/devices/system/cpu/cpu$i/online; done

    Thanks,
    -- Dexuan

    \
     
     \ /
      Last update: 2021-07-15 02:40    [W:2.958 / U:0.340 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site