lkml.org 
[lkml]   [2020]   [Apr]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: WARNING in ib_umad_kill_port
    On Tue, Apr 07, 2020 at 02:39:42PM +0200, Dmitry Vyukov wrote:
    > On Tue, Apr 7, 2020 at 1:55 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
    > >
    > > On Tue, Apr 07, 2020 at 11:56:30AM +0200, Dmitry Vyukov wrote:
    > > > > I'm not sure what could be done wrong here to elicit this:
    > > > >
    > > > > sysfs group 'power' not found for kobject 'umad1'
    > > > >
    > > > > ??
    > > > >
    > > > > I've seen another similar sysfs related trigger that we couldn't
    > > > > figure out.
    > > > >
    > > > > Hard to investigate without a reproducer.
    > > >
    > > > Based on all of the sysfs-related bugs I've seen, my bet would be on
    > > > some races. E.g. one thread registers devices, while another
    > > > unregisters these.
    > >
    > > I did check that the naming is ordered right, at least we won't be
    > > concurrently creating and destroying umadX sysfs of the same names.
    > >
    > > I'm also fairly sure we can't be destroying the parent at the same
    > > time as this child.
    > >
    > > Do you see the above commonly? Could it be some driver core thing? Or
    > > is it more likely something wrong in umad?
    >
    > Mmmm... I can't say, I am looking at some bugs very briefly. I've
    > noticed that sysfs comes up periodically (or was it some other similar
    > fs?).

    Hmm..

    Looking at the git history I see several cases where there are
    ordering problems. I wonder if the rdma parent device is being
    destroyed before the rdma devices complete destruction?

    I see the syzkaller is creating a bunch of virtual net devices, and I
    assume it has created a software rdma device on one of these virtual
    devices.

    So I'm guessing that it is also destroying a parent? But I can't guess
    which.. Some simple tests with veth suggest it is OK because the
    parent is virtual. But maybe bond or bridge or something?

    The issue in rdma is that unregistering a netdev triggers an async
    destruction of the RDMA devices. This has to be async because the
    netdev notification is delivered with RTNL held, and a rdma device
    cannot be destroyed while holding RTNL.

    So there is a race, I suppose, where the netdev can complete
    destruction while rdma continues, and if someone deletes the sysfs
    holding the netdev before rdma completes, I'm going to guess, that we
    hit this warning?

    Could it be? I would love to know what netdev the rdma device was
    created on, but it doesn't seem to show in the trace :\

    This theory could be made more likely by adding a sleep to
    ib_unregister_work() to increase the race window - is there some way
    to get syzkaller to search for a reproducer with that patch?

    Jason

    \
     
     \ /
      Last update: 2020-04-07 16:36    [W:2.578 / U:0.392 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site