Messages in this thread | | | Date | Thu, 24 Aug 2023 13:25:41 +0200 | From | David Hildenbrand <> | Subject | Re: [PATCH RFC 00/37] Add support for arm64 MTE dynamic tag storage reuse |
| |
On 24.08.23 13:06, David Hildenbrand wrote: > On 24.08.23 12:44, Catalin Marinas wrote: >> On Thu, Aug 24, 2023 at 09:50:32AM +0200, David Hildenbrand wrote: >>> after re-reading it 2 times, I still have no clue what your patch set is >>> actually trying to achieve. Probably there is a way to describe how user >>> space intents to interact with this feature, so to see which value this >>> actually has for user space -- and if we are using the right APIs and >>> allocators. >> >> I'll try with an alternative summary, hopefully it becomes clearer (I >> think Alex is away until the end of the week, may not reply >> immediately). If this still doesn't work, maybe we should try a >> different implementation ;). >> >> The way MTE is implemented currently is to have a static carve-out of >> the DRAM to store the allocation tags (a.k.a. memory colour). This is >> what we call the tag storage. Each 16 bytes have 4 bits of tags, so this >> means 1/32 of the DRAM, roughly 3% used for the tag storage. This is >> done transparently by the hardware/interconnect (with firmware setup) >> and normally hidden from the OS. So a checked memory access to location >> X generates a tag fetch from location Y in the carve-out and this tag is >> compared with the bits 59:56 in the pointer. The correspondence from X >> to Y is linear (subject to a minimum block size to deal with some >> address interleaving). The software doesn't need to know about this >> correspondence as we have specific instructions like STG/LDG to location >> X that lead to a tag store/load to Y. >> >> Now, not all memory used by applications is tagged (mmap(PROT_MTE)). >> For example, some large allocations may not use PROT_MTE at all or only >> for the first and last page since initialising the tags takes time. The >> side-effect is that of these 3% DRAM, only part, say 1% is effectively >> used. Some people want the unused tag storage to be released for normal >> data usage (i.e. give it to the kernel page allocator). >> >> So the first complication is that a PROT_MTE page allocation at address >> X will need to reserve the tag storage at location Y (and migrate any >> data in that page if it is in use). >> >> To make things worse, pages in the tag storage/carve-out range cannot >> use PROT_MTE themselves on current hardware, so this adds the second >> complication - a heterogeneous memory layout. The kernel needs to know >> where to allocate a PROT_MTE page from or migrate a current page if it >> becomes PROT_MTE (mprotect()) and the range it is in does not support >> tagging. >> >> Some other complications are arm64-specific like cache coherency between >> tags and data accesses. There is a draft architecture spec which will be >> released soon, detailing how the hardware behaves. >> >> To your question about user APIs/ABIs, that's entirely transparent. As >> with the current kernel (without this dynamic tag storage), a user only >> needs to ask for PROT_MTE mappings to get tagged pages. > > Thanks, that clarifies things a lot. > > So it sounds like you might want to provide that tag memory using CMA. > > That way, only movable allocations can end up on that CMA memory area, > and you can allocate selected tag pages on demand (similar to the > alloc_contig_range() use case). > > That also solves the issue that such tag memory must not be longterm-pinned. > > Regarding one complication: "The kernel needs to know where to allocate > a PROT_MTE page from or migrate a current page if it becomes PROT_MTE > (mprotect()) and the range it is in does not support tagging.", > simplified handling would be if it's in a MIGRATE_CMA pageblock, it > doesn't support tagging. You have to migrate to a !CMA page (for > example, not specifying GFP_MOVABLE as a quick way to achieve that). >
Okay, I now realize that this patch set effectively duplicates some CMA behavior using a new migrate-type. Yeah, that's probably not what we want just to identify if memory is taggable or not.
Maybe there is a way to just keep reusing most of CMA instead.
Another simpler idea to get started would be to just intercept the first PROT_MTE, and allocate all CMA memory. In that case, systems that don't ever use PROT_MTE can have that additional 3% of memory.
You probably know better how frequent it is that only a handful of applications use PROT_MTE, such that there is still a significant portion of tag memory to be reused (and if it's really worth optimizing for that scenario).
-- Cheers,
David / dhildenb
| |