Messages in this thread | | | From | "" <> | Subject | RE: About add an A64FX cache control function into resctrl | Date | Mon, 17 May 2021 08:31:30 +0000 |
| |
Hi Reinette,
I’m sorry for the late reply. I think I could not explain A64FX’s sector cache function well in my first mail. While answering the question, I will also explain this function in more detail. Though maybe you have already learned more about this function by reading specification and manual, in order to better understand this function, some contents may have duplicate explanations.
> >> The overview in section 12 was informative but very high level. > > > > I'm considering how to answer your questions from your email which I > > received before, when I check the email again, I am sorry that the > > information I provided before are insufficient. > > > > To understand the sector cache function of A64FX, could you please see > > A64FX_Microarchitecture_Manual - section 12. Sector Cache > > > https://github.com/fujitsu/A64FX/blob/master/doc/A64FX_Microarchitectu > > re_Manual_en_1.4.pdf > > and, > > A64FX_Specification_HPC_Extension ? section 1.2. Sector Cache > > > https://github.com/fujitsu/A64FX/blob/master/doc/A64FX_Specification_H > > PC_Extension_v1_EN.pdf > > Thank you for the direct links - I missed that there are two documents available. > > After reading the spec portion it does seem to me even more as though > "sectors" could be considered the same as the resctrl "classes of service". The > Fujitsu hardware supports four sectors that can be configured with different > number of ways using the registers you mention above. In resctrl this could be > considered as hardware that supports four classes of service and each class of > service can be allocated a different number of ways.
Fujitsu hardware supports four sectors that can be configured with different number of ways by using "IMP_SCCR" registers, and when this function is added into resctrl, the maximum ways of each sector are indicated by bitmap.
However, A64FX's L2 cache setting registers are shared among PEs (Processor Element) in NUMA. If two PEs in the same NUMA are assigned to different resource groups, changing one PE's L2 setting on one resource group, the other PE's L2 setting on other resource groups will be influenced. So, adding this function into resctrl, we will assign NUMA to the resource group. (On F64FX, each NUMA has 12 PEs, and each PE has L1 cache setting registers, but these registers are not shared.) There are 4 NUMAs on A64FX, 4 NUMAs could be considered as hardware that supports four classes of service at most, and each class of service has 4 sectors (4 L1 sectors& 4 L2 sectors), and each sector can be allocated a different number of ways. And, when a running task on resource group, the [56:57] bits of virtual address are used for sector selection (cache affinity).
> The other part is how hardware knows which sector is being used at any > moment in time. In resctrl that is programmed by writing the active class of > service into needed register at the time the application is context switched > (resctrl_sched_in()). This seems different here since as you describe the > sector is chosen by bits in the address. Even so, which bits to set in the > address needs to be programmed also and I also understand that there is a > "default" sector that can be programmed via register. Could these be equivalent > to what is done currently in resctrl?
Adding this function into resctrl, there is no need to write active class of service into needed register. When running a task, the sector id is decided by [56:57] bits of virtual address, and these bits are programed by users. When creating a resource group, the maximum number of ways of each sector are set by "IMP_SCCR" setting registers. As long as the task is running in a certain resource group, the sector and the maximum number of ways of sectors are used will not be changed. Therefore, we need not consider context switches on A64FX.
> (Could you please also consider my original questions?) I will reply to the original questions mail.
Best regards, Tan Shaopeng
| |