lkml.org 
[lkml]   [2021]   [Nov]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH v3 00/15] Free user PTE page table pages
    On 10.11.21 13:56, Jason Gunthorpe wrote:
    > On Wed, Nov 10, 2021 at 06:54:13PM +0800, Qi Zheng wrote:
    >
    >> In this patch series, we add a pte_refcount field to the struct page of page
    >> table to track how many users of PTE page table. Similar to the mechanism of
    >> page refcount, the user of PTE page table should hold a refcount to it before
    >> accessing. The PTE page table page will be freed when the last refcount is
    >> dropped.
    >
    > So, this approach basically adds two atomics on every PTE map
    >
    > If I have it right the reason that zap cannot clean the PTEs today is
    > because zap cannot obtain the mmap lock due to a lock ordering issue
    > with the inode lock vs mmap lock.

    There are different ways to zap: madvise(DONTNEED) vs
    fallocate(PUNCH_HOLE). It depends on "from where" we're actually
    comming: a process page table walker or the rmap.

    The way locking currently works doesn't allow to remove a page table
    just by holding the mmap lock, not even in write mode. You'll also need
    to hold the respective rmap locks -- which implies that reclaiming apge
    tables crossing VMAs is "problematic". Take a look at khugepaged which
    has to play quite some tricks to remove a page table.

    And there are other ways we can create empty page tables via the rmap,
    like reclaim/writeback, although they are rather a secondary concern mostly.

    >
    > If it could obtain the mmap lock then it could do the zap using the
    > write side as unmapping a vma does.
    >
    > Rather than adding a new "lock" to ever PTE I wonder if it would be
    > more efficient to break up the mmap lock and introduce a specific
    > rwsem for the page table itself, in addition to the PTL. Currently the
    > mmap lock is protecting both the vma list and the page table.

    There is the rmap side of things as well. At least the rmap won't
    reclaim alloc/free page tables, but it will walk page tables while
    holding the respective rmap lock.

    >
    > I think that would allow the lock ordering issue to be resolved and
    > zap could obtain a page table rwsem.
    >
    > Compared to two atomics per PTE this would just be two atomic per
    > page table walk operation, it is conceptually a lot simpler, and would
    > allow freeing all the page table levels, not just PTEs.

    Another alternative is to not do it in the kernel automatically, but
    instead have a madvise(MADV_CLEANUP_PGTABLE) mechanism that will get
    called by user space explicitly once it's reasonable. While this will
    work for the obvious madvise(DONTNEED) users -- like memory allocators
    -- that zap memory, it's a bit more complicated once shared memory is
    involved and we're fallocate(PUNCH_HOLE) memory. But it would at least
    work for many use cases that want to optimize memory consumption for
    sparse memory mappings.

    Note that PTEs are the biggest memory consumer. On x86-64, a 1 TiB area
    will consume 2 GiB of PTE tables and only 4 MiB of PMD tables. So PTEs
    are most certainly the most important part piece.

    --
    Thanks,

    David / dhildenb

    \
     
     \ /
      Last update: 2021-11-10 14:27    [W:4.136 / U:0.016 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site