lkml.org 
[lkml]   [2021]   [Nov]   [9]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH] pci: Don't call resume callback for nearly bound devices
    On Tue, Nov 09, 2021 at 02:05:18PM -0600, Bjorn Helgaas wrote:
    > On Tue, Nov 09, 2021 at 07:58:47PM +0100, Rafael J. Wysocki wrote:
    > > On Tue, Nov 9, 2021 at 7:52 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
    > > > On Tue, Nov 9, 2021 at 7:12 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
    > > > > On Tue, Nov 09, 2021 at 06:18:18PM +0100, Rafael J. Wysocki wrote:
    > > > > > On Tue, Nov 9, 2021 at 7:59 AM Uwe Kleine-König
    > > > > > <u.kleine-koenig@pengutronix.de> wrote:
    > > > > > > On Mon, Nov 08, 2021 at 08:56:19PM -0600, Bjorn Helgaas wrote:
    > > > > > > > [+cc Greg: new device_is_bound() use]
    > > > > > >
    > > > > > > ack, that's what I would have suggested now, too.
    > > > > > >
    > > > > > > > On Mon, Nov 08, 2021 at 10:22:26PM +0100, Uwe Kleine-König wrote:
    > > > > > > > > pci_pm_runtime_resume() exits early when the device to resume isn't
    > > > > > > > > bound yet:
    > > > > > > > >
    > > > > > > > > if (!to_pci_driver(dev->driver))
    > > > > > > > > return 0;
    > > > > > > > >
    > > > > > > > > This however isn't true when the device currently probes and
    > > > > > > > > local_pci_probe() calls pm_runtime_get_sync() because then the driver
    > > > > > > > > core already setup dev->driver. As a result the driver's resume callback
    > > > > > > > > is called before the driver's probe function is called and so more often
    > > > > > > > > than not required driver data isn't setup yet.
    > > > > > > > >
    > > > > > > > > So replace the check for the device being unbound by a check that only
    > > > > > > > > becomes true after .probe() succeeded.
    > > > > > > >
    > > > > > > > I like the fact that this patch is short and simple.
    > > > > > > >
    > > > > > > > But there are 30+ users of to_pci_driver(). This patch asserts that
    > > > > > > > *one* of them, pci_pm_runtime_resume(), is special and needs to test
    > > > > > > > device_is_bound() instead of using to_pci_driver().
    > > > > > >
    > > > > > > Maybe for the other locations using device_is_bound(&pdev->dev) instead
    > > > > > > of to_pci_driver(pdev) != NULL would be nice, too?
    > > > > > >
    > > > > > > I have another doubt: device_is_bound() should (according to its
    > > > > > > kernel-doc) be called with the device lock held. For the call stack that
    > > > > > > is (maybe) fixed here, the lock is held (by __device_attach). We
    > > > > > > probably should check if the lock is also held for the other calls of
    > > > > > > pci_pm_runtime_resume().
    > > > > > >
    > > > > > > Hmm, the device lock is a mutex, the pm functions might be called in
    > > > > > > atomic context, right?
    > > > > > >
    > > > > > > > It's special because the current PM implementation calls it via
    > > > > > > > pm_runtime_get_sync() before the driver's .probe() method. That
    > > > > > > > connection is a little bit obscure and fragile. What if the PM
    > > > > > > > implementation changes?
    > > > > > >
    > > > > > > Maybe a saver bet would be to not use pm_runtime_get_sync() in
    > > > > > > local_pci_probe()?
    > > > > >
    > > > > > Yes, in principle it might be replaced with pm_runtime_get_noresume().
    > > > > >
    > > > > > In theory, that may be problematic if a device is put into a low-power
    > > > > > state on remove and then the driver is bound again to it.
    > > > > >
    > > > > > > I wonder if the same problem exists on remove, i.e. pci_device_remove()
    > > > > > > calls pm_runtime_put_sync() after the driver's .remove() callback was
    > > > > > > called.
    > > > > >
    > > > > > If it is called after ->remove() and before clearing the device's
    > > > > > driver pointer, then yes.
    > > > >
    > > > > Yes, that is the case:
    > > > >
    > > > > pci_device_remove
    > > > > if (drv->remove) {
    > > > > pm_runtime_get_sync
    > > > > drv->remove() # <-- driver ->remove() method
    > > > > pm_runtime_put_noidle
    > > > > }
    > > > > ...
    > > > > pm_runtime_put_sync # <-- after ->remove()
    > > > >
    > > > > So pm_runtime_put_sync() is called after drv->remove(), and it may
    > > > > call drv->pm->runtime_idle(). I think the driver may not expect this.
    > > > >
    > > > > > If this is turned into pm_runtime_put_noidle(), all should work.
    > > > >
    > > > > pci_device_remove() already calls pm_runtime_put_noidle() immediately
    > > > > after calling the driver ->remove() method.
    > > > >
    > > > > Are you saying we should do this, which means pci_device_remove()
    > > > > would call pm_runtime_put_noidle() twice?
    > > >
    > > > Well, they are both needed to keep the PM-runtime reference counting in balance.
    > > >
    > > > This still has an issue, though, because user space would be able to
    > > > trigger a runtime suspend via sysfs after we've dropped the last
    > > > reference to the device in pci_device_remove().
    > > >
    > > > So instead, we can drop the pm_runtime_get_sync() and
    > > > pm_runtime_put_sync() from local_pci_probe() and pci_device_remove(),
    > > > respectively, and add pm_runtine_get_noresume() to pci_pm_init(),
    > > > which will prevent PM-runtime from touching the device until it has a
    > > > driver that supports PM-runtime.
    > > >
    > > > We'll lose the theoretical ability to put unbound devices into D3 this
    > > > way, but we learned some time ago that this isn't safe in all cases
    > > > anyway.
    > >
    > > IOW, something like this (untested and most likely white-space-damaged).
    >
    > Thanks! I applied this manually to for-linus in hopes of making the
    > the next linux-next build.
    >
    > Please send any testing reports and corrections to the patch and
    > commit log!
    >
    > commit dd414877b58b ("PCI/PM: Prevent runtime PM until claimed by a driver that supports it")
    > Author: Bjorn Helgaas <bhelgaas@google.com>
    > Date: Tue Nov 9 13:36:09 2021 -0600
    >
    > PCI/PM: Prevent runtime PM until claimed by a driver that supports it
    >
    > Previously we had a path that could call a driver's ->runtime_resume()
    > method before calling the driver's ->probe() method, which is a problem
    > because ->runtime_resume() often relies on initialization done in
    > ->probe():
    >
    > local_pci_probe
    > pm_runtime_get_sync
    > ...
    > pci_pm_runtime_resume
    > if (!pci_dev->driver)
    > return 0; <-- early exit
    > dev->driver->pm->runtime_resume(); <-- driver ->runtime_resume()
    > pci_dev->driver = pci_drv;
    > pci_drv->probe() <-- driver ->probe()
    >
    > Prior to 2a4d9408c9e8 ("PCI: Use to_pci_driver() instead of
    > pci_dev->driver"), we took the early exit, which avoided the problem. But
    > 2a4d9408c9e8 removed pci_dev->driver (since it's redundant with
    > device->driver), so we no longer take the early exit, which leads to havoc
    > in ->runtime_resume().
    >
    > Similarly, we could call the driver's ->runtime_idle() method after its
    > ->remove() method.
    >
    > Avoid the problem by dropping the pm_runtime_get_sync() and
    > pm_runtime_put_sync() from local_pci_probe() and pci_device_remove(),
    > respectively.
    >
    > Add pm_runtime_get_noresume(), which uses no driver PM callbacks, to the
    > pci_pm_init() enumeration path. This will prevent PM-runtime from touching
    > the device until it has a driver that supports PM-runtime.
    >
    > Link: https://lore.kernel.org/r/CAJZ5v0impb8uscbp8LUTBMExfMoGz=cPrTWhSGh0GF_SANNKPQ@mail.gmail.com
    > Fixes: 2a4d9408c9e8 ("PCI: Use to_pci_driver() instead of pci_dev->driver")
    > Reported-by: Robert Święcki <robert@swiecki.net>
    > Suggested-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
    > Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>

    I like this, this feels better than my initial suggestion using
    device_is_bound().

    Acked-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>

    Thanks
    Uwe

    --
    Pengutronix e.K. | Uwe Kleine-König |
    Industrial Linux Solutions | https://www.pengutronix.de/ |
    [unhandled content-type:application/pgp-signature]
    \
     
     \ /
      Last update: 2021-11-09 21:44    [W:3.400 / U:0.212 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site