lkml.org 
[lkml]   [2024]   [Mar]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH v3] RDMA/cm: add timeout to cm_destroy_id wait
    On Fri, Mar 08, 2024 at 10:33:23PM -0800, Manjunath Patil wrote:
    > Add timeout to cm_destroy_id, so that userspace can trigger any data
    > collection that would help in analyzing the cause of delay in destroying
    > the cm_id.
    >
    > New noinline function helps dtrace/ebpf programs to hook on to it.
    > Existing functionality isn't changed except triggering a probe-able new
    > function at every timeout interval.
    >
    > We have seen cases where CM messages stuck with MAD layer (either due to
    > software bug or faulty HCA), leading to cm_id getting stuck in the
    > following call stack. This patch helps in resolving such issues faster.
    >
    > kernel: ... INFO: task XXXX:56778 blocked for more than 120 seconds.
    > ...
    > Call Trace:
    > __schedule+0x2bc/0x895
    > schedule+0x36/0x7c
    > schedule_timeout+0x1f6/0x31f
    > ? __slab_free+0x19c/0x2ba
    > wait_for_completion+0x12b/0x18a
    > ? wake_up_q+0x80/0x73
    > cm_destroy_id+0x345/0x610 [ib_cm]
    > ib_destroy_cm_id+0x10/0x20 [ib_cm]
    > rdma_destroy_id+0xa8/0x300 [rdma_cm]
    > ucma_destroy_id+0x13e/0x190 [rdma_ucm]
    > ucma_write+0xe0/0x160 [rdma_ucm]
    > __vfs_write+0x3a/0x16d
    > vfs_write+0xb2/0x1a1
    > ? syscall_trace_enter+0x1ce/0x2b8
    > SyS_write+0x5c/0xd3
    > do_syscall_64+0x79/0x1b9
    > entry_SYSCALL_64_after_hwframe+0x16d/0x0
    >
    > Orabug: 36280065

    Not related to the upstream.

    >
    > Signed-off-by: Manjunath Patil <manjunath.b.patil@oracle.com>
    > ---
    > v3:
    > - added #define
    >
    > v2:
    > - removed sysctl related code
    >
    > drivers/infiniband/core/cm.c | 20 +++++++++++++++++++-
    > 1 file changed, 19 insertions(+), 1 deletion(-)
    >
    > diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c
    > index ff58058aeadc..793103cf8152 100644
    > --- a/drivers/infiniband/core/cm.c
    > +++ b/drivers/infiniband/core/cm.c
    > @@ -34,6 +34,7 @@ MODULE_AUTHOR("Sean Hefty");
    > MODULE_DESCRIPTION("InfiniBand CM");
    > MODULE_LICENSE("Dual BSD/GPL");
    >
    > +#define CM_DESTORY_ID_WAIT_TIMEOUT 10000 /* msecs */

    CM_DESTORY_ID_WAIT_TIMEOUT -> CM_DESTROY_ID_WAIT_TIMEOUT

    Fixed and applied.

    Thanks

    \
     
     \ /
      Last update: 2024-05-27 15:43    [W:4.226 / U:0.340 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site