Messages in this thread | | | Date | Thu, 9 Feb 2023 11:34:35 +0530 | Subject | Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing | From | Bharata B Rao <> |
| |
On 2/8/2023 11:42 PM, Dave Hansen wrote: > On 2/8/23 10:03, Peter Zijlstra wrote: >>> - Hardware provided access information could be very useful for driving >>> hot page promotion in tiered memory systems. Need to check if this >>> requires different tuning/heuristics apart from what NUMA balancing >>> already does. >> I think Huang Ying looked at that from the Intel POV and I think the >> conclusion was that it doesn't really work out. What you need is >> frequency information, but the PMU doesn't really give you that. You >> need to process a *ton* of PMU data in-kernel. > > Yeah, there were two big problems. > > First, IIRC, Intel PEBS at the time only gave guest virtual addresses in > the PEBS records. They had to be translated back to host addresses to > be usable. That was extra expensive.
Just to be clear, I am using IBS in host only and it can give both virtual and physical address.
> > Second, it *did* take a lot of processing to turn raw memory accesses > into actionable frequency data. That meant that we started in a hole > performance-wise and had to make *REALLY* good decisions about page > migration to make up for it.
I touched upon the frequency aspect in reply to Peter, but please let me know if I am missing something.
> > The performance data here don't look awful, but they don't seem to add > up to a clear win either. I'm having a hard time imagining who would > turn this on and how widely it would get used in practice.
I am hopeful with more appropriate tuning of NUMA balancing logic to work with hardware-provided access info (as against scan based NUMA hint faults), we should be able to see a clear win. At least theoretically we wouldn't have the overheads of address space scanning and hint faults handling.
Thanks for your inputs.
Regards, Bharata.
| |