lkml.org 
[lkml]   [2013]   [Apr]   [30]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: RFC: IOMMU/AMD: Error Handling
On 4/29/2013 3:10 PM, Don Dutile wrote:
> On 04/29/2013 03:45 PM, Suravee Suthikulanit wrote:
>> Joerg,
>>
>> We are in the process of implementing AMD IOMMU error handling, and I
>> would like some comments from you and the community.
>>
>> Currently, the AMD IOMMU driver only reports events from the event
>> log in the dmesg, and does not try to handle them in case of errors.
>> AMD IOMMU errors can be categorized as device-specific errors and
>> IOMMU errors.
>>
>> 1. For IOMMU errors such as:
>> - DEV_TAB_HADWARE_ERROR
>> - PAGE_TAB_ERROR
>> - COMMAND_HARDWARE_ERROR
>> If the error is detected during IOMMU initialization, we could
>> disable IOMMU and proceed. If the error occurs after IOMMU is
>> initialized, we won't be able to recover from this, and might need to
>> result in panic.
>>
>> 2. For device-specific errors such as:
>> - ILLEGAL_DEV_TABLE_ENTRY
>> - IO_PAGE_FAULT
>> - INVALDE_DEVICE_REQUEST
>> We think the AMD IOMMU driver should try to isolate the device. This
>> involves blocking device transactions at IOMMU DTE and tries to
>> disable the device (e.g. calling the remove(struct pci_dev *pdev)
>> interface generally provides by device drivers). This could prevents
>> the device from continuing to fail and to risk of system instability.
>>
> disabling the device is not an option.
> We've seen mis-configured ACPI tables generate storms
> of invalide dte messages after iommu setup but before they are cleared
> up when
> the OS driver is started & resets the device. The original storm is
> from bios-use
> of IOMMU with a device.
Would some sorts of threshold to help determine the badness of errors
might be sufficient? For instance, if the device has generated N errors,
it is then be removed (where N is tunable through sysfs or kernel boot
options).

> I'd recommend creating a filter that prevents further logging from a
> device
> for 5 mins at a time if a storm of DTE-related errors are seen.
> by definition, the DMA is blocked from corrupting/changing memory, so
> isolation has been established;
> keeping the failure log from consuming the system is the needed fix.

I believe the IOMMU hardware can be configured to suppress logging of subsequent I/O page fault errors until
the device table cache is cleared. This should help avoiding storm of interrupts you are seeing.

>
>> 3. In case of posted memory write transaction, device driver might
>> not be aware that the transaction has failed and blocked at IOMMU. If
>> there is no HW IOMMU, I believe this is handled by PCI error handling
>> code. If the IOMMU hardware reporth such case, could this potentially
>> leverage the Linux IOMMU fault handling interface,
>> iommu_set_fault_handler() and report_iommu_fault(), to communicate to
>> device driver or PCI driver?
>>
> Wondering if you could use AER-like callback mechanism so a driver can
> be invoked when IOMMU error occurs,
> so the device driver can quiesce or reset the device if it deems it
> transient.
That might also be possible. I might need to look into it more.

Suravee
>
>> Any feedback or comments are appreciated.
>>
>> Thank you,
>> Suravee
>>
>>
>>
>>
>> _______________________________________________
>> iommu mailing list
>> iommu@lists.linux-foundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/iommu
>
>




\
 
 \ /
  Last update: 2013-04-30 17:41    [W:0.069 / U:20.108 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site