Messages in this thread |  | | Subject | Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization | From | Tony Krowiak <> | Date | Tue, 21 Aug 2018 15:21:33 -0400 |
| |
On 08/21/2018 12:13 PM, Cornelia Huck wrote: > On Mon, 20 Aug 2018 16:16:15 -0400 > Tony Krowiak <akrowiak@linux.ibm.com> wrote: > >> On 08/20/2018 12:03 PM, Cornelia Huck wrote: >>> On Mon, 13 Aug 2018 17:48:19 -0400 >>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote: >>>> +AP Architectural Overview: >>>> +========================= >>>> +To facilitate the comprehension of the design, let's start with some >>>> +definitions: >>>> + >>>> +* AP adapter >>>> + >>>> + An AP adapter is an IBM Z adapter card that can perform cryptographic >>>> + functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters >>>> + assigned to the LPAR in which a linux host is running will be available to >>>> + the linux host. Each adapter is identified by a number from 0 to 255. When >>>> + installed, an AP adapter is accessed by AP instructions executed by any CPU. >>>> + >>>> + The AP adapter cards are assigned to a given LPAR via the system's Activation >>>> + Profile which can be edited via the HMC. When the system is IPL'd, the AP bus >>> There's lots of s390 jargon in here... but one hopes that someone >>> trying to understand AP is already familiar with the basics... >> I'm not quite sure how one can describe s390-specific devices that can >> be installed >> only on an s390 system without using s390 jargon. I would think that one >> who is >> administering a linux host or guest running on an s390 system would have >> some >> basic knowledge of s390. If you have any suggestions, I'd be happy to >> entertain them. > I fear the jargon is mostly unavoidable :( > >>>> +* AP Instructions: >>>> + >>>> + There are three AP instructions: >>>> + >>>> + * NQAP: to enqueue an AP command-request message to a queue >>>> + * DQAP: to dequeue an AP command-reply message from a queue >>>> + * PQAP: to administer the queues >>> So, NQAP/DQAP need usage domains, while PQAP needs a control domain? Or >>> is it that all of them need usage domains, but PQAP can target a control >>> domain as well? >> All AP instructions - the lone exception being the PQAP(QCI) subfunction - >> identify the usage domain that is the target of the instruction. I think >> using the term 'control domain' is the source of much confusion. It makes >> it seem as if there are two types of domains that serve different purposes. >> That is simply not true. A domain is a partition within an AP adapter that >> can process AP command request messages. All AP commands are sent to a >> domain. Configuring a domain as a usage domain means it can be used to >> process AP commands; in other words, it can be the target of an AP >> instruction. Configuring a domain as a control domain means it can be >> changed by an AP command. AP commands that change a domain are sent to >> a usage domain, but the domain to be changed is specified in the payload >> of the AP command message. The domain thus specified must be >> identified via the AP configuration as a control domain, or the AP command >> will be rejected. > Yes, the 'control domain' term is a source of much confusion :( > >>> [I don't want to dive deeply into the AP architecture here, just far >>> enough to really understand the design implications.] >> Are you suggesting some of the above should be removed? If so, what? > Not removed. What about an explanation like the following somewhere: > > "AP instructions identify the domain that is targeted to process the > command: This must be one of the usage domains. They may modify a > domain that is not one of the usage domains, but the modified domain > must be one of the control domains." > > I hope that is both correct and understandable ;)
Yes, it is both correct and understandable.
> >>> Does the SIE complain if you specify a control >>> domain that the host does not have access to (I'd guess so)? >> The SIE does not complain if you specify a domain to which the host - or a >> lower level guest - does not have access. The firmware performs a logical >> AND of the guest's and hosts's - or lower level guest's - APMs, AQMs and >> ADMs >> to create effective masks EAPM, EAQM and EADM. Only devices corresponding to >> the bits set in the EAPM, EAQM and EADM will be accessible by the guest. > OK, so the guest effectively won't see the domain. That makes sense.
It is one of the positive aspects of the architecture.
> >>> >>>> + >>>> +The APQNs can provide secure key functionality - i.e., a private key is stored >>>> +on the adapter card for each of its domains - so each APQN must be assigned to >>>> +at most one guest or to the linux host. >>>> + >>>> + Example 1: Valid configuration: >>>> + ------------------------------ >>>> + Guest1: adapters 1,2 domains 5,6 >>>> + Guest2: adapter 1,2 domain 7 >>>> + >>>> + This is valid because both guests have a unique set of APQNs: Guest1 has >>>> + APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and (2,7). >>>> + >>>> + Example 2: Invalid configuration: >>>> + Guest1: adapters 1,2 domains 5,6 >>>> + Guest2: adapter 1 domains 6,7 >>>> + >>>> + This is an invalid configuration because both guests have access to >>>> + APQN (1,6). >>> So, the adapters or the domains can overlap , but the cross product >>> mustn't? If I had >>> >>> Guest1: adapters 1,2 domains 5,6 >>> Guest2: adapters 3,4 domains 5,6 >>> >>> would that be fine? >> Yes, that would be fine because Guest1 would have access to APQNs >> (1,5), (1,6), (2,5) and (2,6) while Guest2 would have access to >> (3,5), (3,6), (4,5) AND (4,6), but neither would have access to >> the same APQN. > Might be a good idea to add this as an additional example.
Will do
> >>> Is there any rule about shared control domains? >> AFAIK there isn't, but I will consult with Reinhard about that. >> >>> (...) >>> >>>> +Limitations >>>> +=========== >>>> +* The KVM/kernel interfaces do not provide a way to prevent unbinding an AP >>>> + queue that is still assigned to a mediated device. Even if the device >>>> + 'remove' callback returns an error, the device core detaches the AP >>>> + queue from the VFIO AP driver. It is therefore incumbent upon the >>>> + administrator to make sure there is no mediated device to which the >>>> + APQN - for the AP queue being unbound - is assigned. >>>> + >>>> +* Hot plug/unplug of AP devices is not supported for guests. >>> Not sure what that sentence means. Adding/removing devices by the >>> hypervisor is not supported? Or some guest actions, respectively >>> injecting notifications that would trigger some actions on the real >>> hardware? >> No means is provided to modify a guest's AP matrix - i.e., APM, AQM >> and ADM - while a guest is running. Once a guest is running, its AP >> configuration can not be changed dynamically. >> >>> Do you want to add (some of) this in the future? >> Yes, we plan to introduce dynamic configurations in future releases. > What about the following sentence: > > "Dynamically modifying the AP matrix for a running guest (which would > amount to hot(un)plug of AP devices for the guest) is currently not > supported."
Sounds fine to me.
> >>> >>>> + >>>> +* Live guest migration is not supported for guests using AP devices. >>> Migration and vfio is an interesting area in general :) Would be great >>> if vfio-ap could benefit from any generic efforts in that area, but >>> that probably requires that someone with access to documentation and >>> hardware keeps an eye on developments. >> I have briefly looked at some of the articles talking about live migration >> of passthrough devices, but nothing seemed applicable to AP architecture. > Most of the approaches to live migration of vfio devices are focused on > pci devices; even ccw devices have different needs. Any halfway generic > approach would need a common part and a backend-specific part anyway, I > think.
Yes, that would seem to be the case.
> >> From my limited perspective, it would seem that architectural changes >> would have to be implemented to fully support live migration of in-process >> AP queues. > From what I have seen of the AP virtualization architecture, this may > very well be the case. I'll keep AP in the back of my head, but it's > probably better to focus on the easier targets first.
That has been our goal from the start.
>
|  |