[lkml]   [2023]   [Feb]   [21]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [PATCH v2 09/46] mm: add MADV_SPLIT to enable HugeTLB HGM
    On Fri, Feb 17, 2023 at 5:58 PM Mina Almasry <> wrote:
    > On Fri, Feb 17, 2023 at 4:28 PM James Houghton <> wrote:
    > >
    > > Issuing ioctl(MADV_SPLIT) on a HugeTLB address range will enable
    > > HugeTLB HGM. MADV_SPLIT was chosen for the name so that this API can be
    > > applied to non-HugeTLB memory in the future, if such an application is
    > > to arise.
    > >
    > > MADV_SPLIT provides several API changes for some syscalls on HugeTLB
    > > address ranges:
    > > 1. UFFDIO_CONTINUE is allowed for MAP_SHARED VMAs at PAGE_SIZE
    > > alignment.
    > > 2. read()ing a page fault event from a userfaultfd will yield a
    > > PAGE_SIZE-rounded address, instead of a huge-page-size-rounded
    > > address (unless UFFD_FEATURE_EXACT_ADDRESS is used).
    > >
    > > There is no way to disable the API changes that come with issuing
    > > MADV_SPLIT. MADV_COLLAPSE can be used to collapse high-granularity page
    > > table mappings that come from the extended functionality that comes with
    > > using MADV_SPLIT.
    > >
    > So is a hugetlb page or VMA that has been MADV_SPLIT + MADV_COLLAPSE
    > distinct from a hugetlb page or vma that has not been? I thought
    > COLLAPSE would reverse the effects on SPLIT completely.

    Right now, MADV_COLLAPSE does *not* completely undo the effects of an
    MADV_SPLIT. The API changes that come from MADV_SPLIT aren't undone
    with an MADV_COLLAPSE.

    > > For post-copy live migration, the expected use-case is:
    > > 1. mmap(MAP_SHARED, some_fd) primary mapping
    > > 2. mmap(MAP_SHARED, some_fd) alias mapping
    > > 3. MADV_SPLIT the primary mapping
    > > 4. UFFDIO_REGISTER/etc. the primary mapping
    > > 5. Copy memory contents into alias mapping and UFFDIO_CONTINUE the
    > > corresponding PAGE_SIZE sections in the primary mapping.
    > >
    > Huh, so MADV_SPLIT doesn't actually split an existing PMD mapping into
    > high granularity mappings. Instead it says that future mappings may be
    > high granularity? I assume they may not even be high granularity, like
    > if the alias mapping faulted in a full hugetlb page (without
    > UFFDIO_CONTINUE) that page would be regular mapped not high
    > granularity mapped.

    MADV_SPLIT just means "userspace is aware that they are able to start
    mapping HugeTLB pages at high-granularity". Right now the only way to
    get high-granularity mappings is with UFFDIO_CONTINUE, but there may
    be other ways in the future.

    As of this series, if you MADV_SPLIT a HugeTLB VMA and you aren't
    using userfaultfd minor faults, it's basically a no-op. The mappings
    that are created will still be huge. I could change this, but I don't
    really see a reason to right now.

    > This may be bikeshedding but I do think a clearer name is warranted.
    > Maybe MADV_MAY_SPLIT or something.

    I agree -- MADV_MAY_SPLIT more accurately describes the HugeTLB
    functionality. I really don't mind what the MADV is called.

    I think enabling the high-granularity userfaultfd bits with a
    userfaultfd feature[1] worked reasonably well. There is some API
    discussion in that thread[1].


     \ /
      Last update: 2023-03-27 00:30    [W:2.259 / U:0.056 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site