lkml.org 
[lkml]   [2021]   [Jun]   [8]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [RFC] /dev/ioasid uAPI proposal
    On Thu, Jun 03, 2021 at 09:28:32AM -0300, Jason Gunthorpe wrote:
    > On Thu, Jun 03, 2021 at 03:23:17PM +1000, David Gibson wrote:
    > > On Wed, Jun 02, 2021 at 01:37:53PM -0300, Jason Gunthorpe wrote:
    > > > On Wed, Jun 02, 2021 at 04:57:52PM +1000, David Gibson wrote:
    > > >
    > > > > I don't think presence or absence of a group fd makes a lot of
    > > > > difference to this design. Having a group fd just means we attach
    > > > > groups to the ioasid instead of individual devices, and we no longer
    > > > > need the bookkeeping of "partial" devices.
    > > >
    > > > Oh, I think we really don't want to attach the group to an ioasid, or
    > > > at least not as a first-class idea.
    > > >
    > > > The fundamental problem that got us here is we now live in a world
    > > > where there are many ways to attach a device to an IOASID:
    > >
    > > I'm not seeing that that's necessarily a problem.
    > >
    > > > - A RID binding
    > > > - A RID,PASID binding
    > > > - A RID,PASID binding for ENQCMD
    > >
    > > I have to admit I haven't fully grasped the differences between these
    > > modes. I'm hoping we can consolidate at least some of them into the
    > > same sort of binding onto different IOASIDs (which may be linked in
    > > parent/child relationships).
    >
    > What I would like is that the /dev/iommu side managing the IOASID
    > doesn't really care much, but the device driver has to tell
    > drivers/iommu what it is going to do when it attaches.

    By the device driver, do you mean the userspace or guest device
    driver? Or do you mean the vfio_pci or mdev "shim" device driver"?

    > It makes sense, in PCI terms, only the driver knows what TLPs the
    > device will generate. The IOMMU needs to know what TLPs it will
    > recieve to configure properly.
    >
    > PASID or not is major device specific variation, as is the ENQCMD/etc
    >
    > Having the device be explicit when it tells the IOMMU what it is going
    > to be sending is a major plus to me. I actually don't want to see this
    > part of the interface be made less strong.

    Ok, if I'm understanding this right a PASID capable IOMMU will be able
    to process *both* transactions with just a RID and transactions with a
    RID+PASID.

    So if we're thinking of this notional 84ish-bit address space, then
    that includes "no PASID" as well as all the possible PASID values.
    Yes? Or am I confused?

    >
    > > > The selection of which mode to use is based on the specific
    > > > driver/device operation. Ie the thing that implements the 'struct
    > > > vfio_device' is the thing that has to select the binding mode.
    > >
    > > I thought userspace selected the binding mode - although not all modes
    > > will be possible for all devices.
    >
    > /dev/iommu is concerned with setting up the IOAS and filling the IO
    > page tables with information
    >
    > The driver behind "struct vfio_device" is responsible to "route" its
    > HW into that IOAS.
    >
    > They are two halfs of the problem, one is only the io page table, and one
    > the is connection of a PCI TLP to a specific io page table.
    >
    > Only the driver knows what format of TLPs the device will generate so
    > only the driver can specify the "route"

    Ok. I'd really like if we can encode this in a way that doesn't build
    PCI-specific structure into the API, though.

    >
    > > > eg if two PCI devices are in a group then it is perfectly fine that
    > > > one device uses RID binding and the other device uses RID,PASID
    > > > binding.
    > >
    > > Uhhhh... I don't see how that can be. They could well be in the same
    > > group because their RIDs cannot be distinguished from each other.
    >
    > Inability to match the RID is rare, certainly I would expect any IOMMU
    > HW that can do PCIEe PASID matching can also do RID matching.

    It's not just up to the IOMMU. The obvious case is a PCIe-to-PCI
    bridge. All transactions show the RID of the bridge, because vanilla
    PCI doesn't have them. Same situation with a buggy multifunction
    device which uses function 0's RID for all functions.

    It may be rare, but we still have to deal with it one way or another.

    I really don't think we want to support multiple binding types for a
    single group.

    > With
    > such HW the above is perfectly fine - the group may not be secure
    > between members (eg !ACS), but the TLPs still carry valid RIDs and
    > PASID and the IOMMU can still discriminate.

    They carry RIDs, whether they're valid depends on how buggy your
    hardware is.

    > I think you are talking about really old IOMMU's that could only
    > isolate based on ingress port or something.. I suppose modern PCIe has
    > some cases like this in the NTB stuff too.

    Depends what you mean by really old. They may seem really old to
    those working on new fancy IOMMU technology. But I hit problems in
    practice not long ago with awkwardly multi-device groups because it
    was on a particular Dell server without ACS implementation. Likewise
    I strongly suspect non-PASID IOMMUs will remain common on low end
    hardware (like peoples' laptops) for some time.

    > Oh, I hadn't spent time thinking about any of those.. It is messy but
    > it can still be forced to work, I guess. A device centric model means
    > all the devices using the same routing ID have to be connected to the
    > same IOASID by userspace. So some of the connections will be NOPs.

    See, that's exactly what I thought the group checks were enforcing.
    I'm really hoping we don't need two levels of granularity here: groups
    of devices that can't be identified from each other, and then groups
    of those that can't be isolated from each other. That introduces a
    huge amount of extra conceptual complexity.

    --
    David Gibson | I'll have my music baroque, and my code
    david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
    | _way_ _around_!
    http://www.ozlabs.org/~dgibson
    [unhandled content-type:application/pgp-signature]
    \
     
     \ /
      Last update: 2021-06-08 08:55    [W:4.627 / U:0.236 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site