Messages in this thread | ![/](/images/icornerl.gif) | | Subject | Re: About add an A64FX cache control function into resctrl | From | Reinette Chatre <> | Date | Wed, 21 Jul 2021 16:39:25 -0700 |
| |
Hi Tan Shaopeng,
On 7/21/2021 1:10 AM, tan.shaopeng@fujitsu.com wrote: > Hi Reinette, > >> On 7/7/2021 4:26 AM, tan.shaopeng@fujitsu.com wrote: >>>>> Sorry, I have not explained A64FX's sector cache function well yet. >>>>> I think I need explain this function from different perspective. >>>> >>>> You have explained the A64FX's sector cache function well. I have >>>> also read both specs to understand it better. It appears to me that >>>> you are not considering the resctrl architecture as part of your >>>> solution but instead just forcing your architecture onto the resctrl >>>> filesystem. For example, in resctrl the resource groups are not just >>>> a directory structure but has significance in what is being >>>> represented within the directory (a class of service). The files >>>> within a resource group's directory build on that. From your side I >>>> have not seen any effort in aligning the sector cache function with the >> resctrl architecture but instead you are just changing resctrl interface to match >> the A64FX architecture. >>>> >>>> Could you please take a moment to understand what resctrl is and how >>>> it could be mapped to A64FX in a coherent way? >>> >>> Previously, my idea is based on how to make instructions use different >>> sectors in one task. After I studied resctrl, to utilize resctrl >>> architecture on A64FX, I think it’s better to assign one sector to one >>> task. Thanks for your idea that "sectors" could be considered the same >>> as the resctrl "classes of service". >>> >>> Based on your idea, I am considering the implementation details. >>> In this email, I will explain the outline of new proposal, and then >>> please allow me to confirm a few technologies about resctrl. >>> >>> The outline of my proposal is as follows. >>> - Add a sector function equivalent to Intel's CAT function into resctrl. >>> (divide shared L2 cache into multiple partitions for multiple cores >>> use) >>> - Allocate one sector to one resource group (one CLOSID). Since one >>> core can only be assigned to one resource group, on A64FX each core >>> only uses one sector at a time. >> >> ok, so a sector is a portion of cache and matches with what can be represented >> with a resource group. >> >> The second part of your comment is not clear to me. In the first part you >> mention: "one core can only be assigned to one resource group" - this seems to >> indicate some static assignment between cores and sectors and if this is the > > Sorry, does "static assignment between cores and sectors" mean > each core always use a fixed sector id? For example, core 0 always > use sector 0 at any case. It is not. > >> case this needs more thinking since the current implementation assumes that >> any core that can access the cache can access all resource groups associated >> with that cache. On the other hand, you mention "on A64FX each core only uses >> one sector at a time" - this now sounds dynamic and is how resctrl works since >> the CPU is assigned a single class of service to indicate all resources >> accessible to it. > > It is correct. Each core can be assigned to any resource group, and > each core only uses one sector at a time. Additionally, which sector > each core uses depends on the resource group (class of service) ID.
Thank you for clarifying. From what I understand this could be supported by existing resctrl flows.
...
>>> In resctrl_sched_in(), there are comments as follow: >>> /* >>> * If this task has a closid/rmid assigned, use it. >>> * Else use the closid/rmid assigned to this cpu. >>> */ >>> I thought when we write PID to tasks file, this task (PID) will only >>> run on the CPUs which are specified in cpus file in the same resource >>> group. So, the task_struct's closid and cpu's closid is the same. >>> When task's closid is different from cpu's closid? >> >> resctrl does not manage the affinity of tasks. >> >> Tony recently summarized the cpus file very well to me: The actual semantics of >> the CPUs file is to associate a CLOSid for a task that is in the default resctrl >> group ? while it is running on one of the listed CPUs. >> >> To answer your question the task's closid could be different from the CPU's >> closid if the task's closid is 0 while it is running on a CPU that is in the cpus file >> of a non-default resource group. >> >> You can see a summary of the decision flow in section "Resource allocation >> rules" in Documentation/x86/resctrl.rst >> >> The "cpus" file was created in support of the real-time use cases. In these use >> cases a group of CPUs can be designated as supporting the real-time work and >> with their own resource group and assigned the needed resources to do the >> real-time work. A real-time task can then be started with affinity to those CPUs >> and dynamically any kernel threads (that will be started on the same CPU) >> doing work on behalf of this task would be able to use the resources set aside >> for the real-time work. > > Thanks for your explanation. I understood it. > > I will implement this sector function, and if I have other questions, > please allow me to mail you.
I will help where I can. You may also be interested in the work James is busy with. See his latest series at https://lore.kernel.org/lkml/20210715173043.14222-1-james.morse@arm.com/
Reinette
| ![\](/images/icornerr.gif) |