lkml.org 
[lkml]   [2020]   [Oct]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: [PATCH v7 0/5] Fix DPC hotplug race and enhance error handling
Raj,

On Sun, Oct 4, 2020 at 12:57 PM Raj, Ashok <ashok.raj@linux.intel.com> wrote:
>
> Hi Ethan
>
> On Sat, Oct 03, 2020 at 03:55:09AM -0400, Ethan Zhao wrote:
> > Hi,folks,
> >
> > This simple patch set fixed some serious security issues found when DPC
> > error injection and NVMe SSD hotplug brute force test were doing -- race
> > condition between DPC handler and pciehp, AER interrupt handlers, caused
> > system hang and system with DPC feature couldn't recover to normal
> > working state as expected (NVMe instance lost, mount operation hang,
> > race PCIe access caused uncorrectable errors reported alternatively etc).
>
> I think maybe picking from other commit messages to make this description in
> cover letter bit clear. The fundamental premise is that when due to error
> conditions when events are processed by both DPC handler and hotplug handling of
> DLLSC both operating on the same device object ends up with crashes.

Yep, that's right.

Thanks,
Ethan
>
>
> >
> > With this patch set applied, stable 5.9-rc6 on ICS (Ice Lake SP platform,
> > see
> > https://en.wikichip.org/wiki/intel/microarchitectures/ice_lake_(server))
> >
> > could pass the PCIe Gen4 NVMe SSD brute force hotplug test with any time
> > interval between hot-remove and plug-in operation tens of times without
> > any errors occur and system works normal.
>
> >
> > With this patch set applied, system with DPC feature could recover from
> > NON-FATAL and FATAL errors injection test and works as expected.
> >
> > System works smoothly when errors happen while hotplug is doing, no
> > uncorrectable errors found.
> >
> > Brute DPC error injection script:
> >
> > for i in {0..100}
> > do
> > setpci -s 64:02.0 0x196.w=000a
> > setpci -s 65:00.0 0x04.w=0544
> > mount /dev/nvme0n1p1 /root/nvme
> > sleep 1
> > done
> >
> > Other details see every commits description part.
> >
> > This patch set could be applied to stable 5.9-rc6/rc7 directly.
> >
> > Help to review and test.
> >
> > v2: changed according to review by Andy Shevchenko.
> > v3: changed patch 4/5 to simpler coding.
> > v4: move function pci_wait_port_outdpc() to DPC driver and its
> > declaration to pci.h. (tip from Christoph Hellwig <hch@infradead.org>).
> > v5: fix building issue reported by lkp@intel.com with some config.
> > v6: move patch[3/5] as the first patch according to Lukas's suggestion.
> > and rewrite the comment part of patch[3/5].
> > v7: change the patch[4/5], based on Bjorn's code and truth table.
> > change the patch[5/5] about the debug output information.
> >
> > Thanks,
> > Ethan
> >
> >
> > Ethan Zhao (5):
> > PCI/ERR: get device before call device driver to avoid NULL pointer
> > dereference
> > PCI/DPC: define a function to check and wait till port finish DPC
> > handling
> > PCI: pciehp: check and wait port status out of DPC before handling
> > DLLSC and PDC
> > PCI: only return true when dev io state is really changed
> > PCI/ERR: don't mix io state not changed and no driver together
> >
> > drivers/pci/hotplug/pciehp_hpc.c | 4 ++-
> > drivers/pci/pci.h | 55 +++++++++++++-------------------
> > drivers/pci/pcie/dpc.c | 27 ++++++++++++++++
> > drivers/pci/pcie/err.c | 18 +++++++++--
> > 4 files changed, 68 insertions(+), 36 deletions(-)
> >
> >
> > base-commit: a1b8638ba1320e6684aa98233c15255eb803fac7
> > --
> > 2.18.4
> >

\
 
 \ /
  Last update: 2020-10-07 09:34    [W:0.397 / U:0.024 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site