Messages in this thread | | | Date | Fri, 21 Oct 2022 13:09:25 -0700 | Subject | Re: [RFD] resctrl: reassigning a running container's CTRL_MON group | From | Reinette Chatre <> |
| |
Hi James,
On 10/19/2022 6:57 AM, James Morse wrote: > Hi Peter, > > On 17/10/2022 11:15, Peter Newman wrote: >> On Wed, Oct 12, 2022 at 6:55 PM James Morse <james.morse@arm.com> wrote: >>> You originally asked: >>> | Any concerns about the CLOSID-reusing behavior? >>> >>> I don't think this will work well with MPAM ... I expect it will mess up the bandwidth >>> counters. >>> >>> MPAM's equivalent to RMID is PMG. While on x86 CLOSID and RMID are independent numbers, >>> this isn't true for PARTID (MPAM's version of CLOSID) and PMG. The PMG bits effectively >>> extended the PARTID with bits that aren't used to look up the configuration. >>> >>> x86's monitors match only on RMID, and there are 'enough' RMID... MPAMs monitors are more >>> complicated. I've seen details of a system that only has 1 bit of PMG space. >>> >>> While MPAM's bandwidth monitors can match just the PMG, there aren't expected to be enough >>> unique PMG for every control/monitor group to have a unique value. Instead, MPAM's >>> monitors are expected to be used with both the PARTID and PMG. >>> >>> ('bandwidth monitors' is relevant here, MPAM's 'cache storage utilisation' monitors can't >>> match on just PMG at all - they have to be told the PARTID too) >>> >>> >>> If you're re-using CLOSID like this, I think you'll end up with noisy measurements on MPAM >>> systems as the caches hold PARTID/PMG values from before the re-use pattern changed, and >>> the monitors have to match on both. > >> Yes, that sounds like it would be an issue. >> >> Following your refactoring changes, hopefully the MPAM driver could >> offer alternative methods for managing PARTIDs and PMGs depending on the >> available hardware resources. > > Mmmm, I don't think anything other than one-partid per control group and one-pmg per > monitor group makes much sense. > > >> If there are a lot more PARTIDs than PMGs, then it would fit well with a >> user who never creates child MON groups. In case the number of MON >> groups gets ahead of the number of CTRL_MON groups and you've run out of >> PMGs, perhaps you would just try to allocate another PARTID and program >> the same partitioning configuration before giving up. > > User-space can choose to do this. > If the kernel tries to be clever and do this behind user-space's back, it needs to > allocate two monitors for this secretly-two-control-groups, and always sum the counters > before reporting them to user-space.
If I understand this scenario correctly, the kernel is already doing this. As implemented in mon_event_count() the monitor data of a CTRL_MON group is the sum of the parent CTRL_MON group and all its child MON groups.
> If monitors are a contended resource, then you may be unable to monitor the > secretly-two-control-groups group once the kernel has done this.
I am not viewing this as "secretly-two-control-groups" - there would still be only one parent CTRL_MON group that dictates all the allocations. MON groups already have a CLOSID (PARTID) property but at this time it is always identical to the parent CTRL_MON group. The difference introduced is that some of the child MON groups may have a different CLOSID (PARTID) from the parent.
> > I don't think the kernel should try to be too clever here. >
That is a fair concern but it may be worth exploring as it seems to address a few ABI concerns and user space seems to be eyeing using a future "num_closid" info as a check of "RDT/PQoS" vs "MPAM".
Reinette
| |