lkml.org 
[lkml]   [2023]   [Sep]   [15]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    SubjectRe: [PATCH v2 6/6] fork: Use __mt_dup() to duplicate maple tree in dup_mmap()
    From


    在 2023/9/15 18:51, Peng Zhang 写道:
    >
    >
    > 在 2023/9/8 04:14, Liam R. Howlett 写道:
    >> * Peng Zhang <zhangpeng.00@bytedance.com> [230830 08:58]:
    >>> Use __mt_dup() to duplicate the old maple tree in dup_mmap(), and then
    >>> directly modify the entries of VMAs in the new maple tree, which can
    >>> get better performance. The optimization effect is proportional to the
    >>> number of VMAs.
    >>>
    >>> There is a "spawn" in byte-unixbench[1], which can be used to test the
    >>> performance of fork(). I modified it slightly to make it work with
    >>> different number of VMAs.
    >>>
    >>> Below are the test numbers. There are 21 VMAs by default. The first row
    >>> indicates the number of added VMAs. The following two lines are the
    >>> number of fork() calls every 10 seconds. These numbers are different
    >>> from the test results in v1 because this time the benchmark is bound to
    >>> a CPU. This way the numbers are more stable.
    >>>
    >>>    Increment of VMAs: 0      100     200     400     800     1600
    >>> 3200    6400
    >>> 6.5.0-next-20230829: 111878 75531   53683   35282   20741   11317
    >>> 6110    3158
    >>> Apply this patchset: 114531 85420   64541   44592   28660   16371
    >>> 9038    4831
    >>>                       +2.37% +13.09% +20.23% +26.39% +38.18% +44.66%
    >>> +47.92% +52.98%
    >>
    >> Thanks!
    >>
    >> Can you include 21 in this table since it's the default?
    >>
    >>>
    >>> [1] https://github.com/kdlucas/byte-unixbench/tree/master
    >>>
    >>> Signed-off-by: Peng Zhang <zhangpeng.00@bytedance.com>
    >>> ---
    >>>   kernel/fork.c | 34 ++++++++++++++++++++++++++--------
    >>>   mm/mmap.c     | 14 ++++++++++++--
    >>>   2 files changed, 38 insertions(+), 10 deletions(-)
    >>>
    >>> diff --git a/kernel/fork.c b/kernel/fork.c
    >>> index 3b6d20dfb9a8..e6299adefbd8 100644
    >>> --- a/kernel/fork.c
    >>> +++ b/kernel/fork.c
    >>> @@ -650,7 +650,6 @@ static __latent_entropy int dup_mmap(struct
    >>> mm_struct *mm,
    >>>       int retval;
    >>>       unsigned long charge = 0;
    >>>       LIST_HEAD(uf);
    >>> -    VMA_ITERATOR(old_vmi, oldmm, 0);
    >>>       VMA_ITERATOR(vmi, mm, 0);
    >>>       uprobe_start_dup_mmap();
    >>> @@ -678,17 +677,39 @@ static __latent_entropy int dup_mmap(struct
    >>> mm_struct *mm,
    >>>           goto out;
    >>>       khugepaged_fork(mm, oldmm);
    >>> -    retval = vma_iter_bulk_alloc(&vmi, oldmm->map_count);
    >>> -    if (retval)
    >>> +    /* Use __mt_dup() to efficiently build an identical maple tree. */
    >>> +    retval = __mt_dup(&oldmm->mm_mt, &mm->mm_mt, GFP_NOWAIT |
    >>> __GFP_NOWARN);
    >>
    >> Apparently the flags should be GFP_KERNEL here so that compaction can
    >> run.
    >>
    >>> +    if (unlikely(retval))
    >>>           goto out;
    >>>       mt_clear_in_rcu(vmi.mas.tree);
    >>> -    for_each_vma(old_vmi, mpnt) {
    >>> +    for_each_vma(vmi, mpnt) {
    >>>           struct file *file;
    >>>           vma_start_write(mpnt);
    >>>           if (mpnt->vm_flags & VM_DONTCOPY) {
    >>>               vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt));
    >>> +
    >>> +            /*
    >>> +             * Since the new tree is exactly the same as the old one,
    >>> +             * we need to remove the unneeded VMAs.
    >>> +             */
    >>> +            mas_store(&vmi.mas, NULL);
    >>> +
    >>> +            /*
    >>> +             * Even removing an entry may require memory allocation,
    >>> +             * and if removal fails, we use XA_ZERO_ENTRY to mark
    >>> +             * from which VMA it failed. The case of encountering
    >>> +             * XA_ZERO_ENTRY will be handled in exit_mmap().
    >>> +             */
    >>> +            if (unlikely(mas_is_err(&vmi.mas))) {
    >>> +                retval = xa_err(vmi.mas.node);
    >>> +                mas_reset(&vmi.mas);
    >>> +                if (mas_find(&vmi.mas, ULONG_MAX))
    >>> +                    mas_store(&vmi.mas, XA_ZERO_ENTRY);
    >>> +                goto loop_out;
    >>> +            }
    >>> +
    >>
    >> Storing NULL may need extra space as you noted, so we need to be careful
    >> what happens if we don't have that space.  We should have a testcase to
    >> test this scenario.
    >>
    >> mas_store_gfp() should be used with GFP_KERNEL.  The VMAs use GFP_KERNEL
    >> in this function, see vm_area_dup().
    >>
    >> Don't use the exit_mmap() path to undo a failed fork.  You've added
    >> checks and complications to the exit path for all tasks in the very
    >> unlikely event that we run out of memory when we hit a very unlikely
    >> VM_DONTCOPY flag.
    >>
    >> I see the issue with having a portion of the tree with new VMAs that are
    >> accounted and a portion of the tree that has old VMAs that should not be
    >> looked at.  It was clever to use the XA_ZERO_ENTRY as a stop point, but
    >> we cannot add that complication to the exit path and then there is the
    >> OOM race to worry about (maybe, I am not sure since this MM isn't
    >> active yet).
    > I encountered some errors after implementing the scheme you mentioned
    > below. This would also clutter fork.c and mmap.c, as some internal
    > functions would need to be made global.
    >
    > I thought of another way to put everything into maple tree. In non-RCU
    > mode, we can remove the last half of the tree without allocating any
    > memory. This requires modifications to the internal implementation of
    > mas_store().
    > Then remove the second half of the tree like this:
    >
    > mas.index = 0;
    Sorry, typo.
    Change to: mas.index = vma->start
    > mas.last = ULONGN_MAX;
    > mas_store(&mas, NULL).

    >
    > At least in non-RCU mode, we can do this, since we only need to merge
    > some nodes, or move some items to adjacent nodes.
    > However, this will increase the workload significantly.
    >
    >>
    >> Using what is done in exit_mmap() and do_vmi_align_munmap() as a
    >> prototype, we can do something like the *untested* code below:
    >>
    >> if (unlikely(mas_is_err(&vmi.mas))) {
    >>     unsigned long max = vmi.index;
    >>
    >>     retval = xa_err(vmi.mas.node);
    >>     mas_set(&vmi.mas, 0);
    >>     tmp = mas_find(&vmi.mas, ULONG_MAX);
    >>     if (tmp) { /* Not the first VMA failed */
    >>         unsigned long nr_accounted = 0;
    >>
    >>         unmap_region(mm, &vmi.mas, vma, NULL, mpnt, 0, max, max,
    >>                 true);
    >>         do {
    >>             if (vma->vm_flags & VM_ACCOUNT)
    >>                 nr_accounted += vma_pages(vma);
    >>             remove_vma(vma, true);
    >>             cond_resched();
    >>             vma = mas_find(&vmi.mas, max - 1);
    >>         } while (vma != NULL);
    >>
    >>         vm_unacct_memory(nr_accounted);
    >>     }
    >>     __mt_destroy(&mm->mm_mt);
    >>     goto loop_out;
    >> }
    >>
    >> Once exit_mmap() is called, the check for OOM (no vma) will catch that
    >> nothing is left to do.
    >>
    >> It might be worth making an inline function to do this to keep the fork
    >> code clean.  We should test this by detecting a specific task name and
    >> returning a failure at a given interval:
    >>
    >> if (!strcmp(current->comm, "fork_test") {
    >> ...
    >> }
    >>
    >>
    >>>               continue;
    >>>           }
    >>>           charge = 0;
    >>> @@ -750,8 +771,7 @@ static __latent_entropy int dup_mmap(struct
    >>> mm_struct *mm,
    >>>               hugetlb_dup_vma_private(tmp);
    >>>           /* Link the vma into the MT */
    >>> -        if (vma_iter_bulk_store(&vmi, tmp))
    >>> -            goto fail_nomem_vmi_store;
    >>> +        mas_store(&vmi.mas, tmp);
    >>>           mm->map_count++;
    >>>           if (!(tmp->vm_flags & VM_WIPEONFORK))
    >>> @@ -778,8 +798,6 @@ static __latent_entropy int dup_mmap(struct
    >>> mm_struct *mm,
    >>>       uprobe_end_dup_mmap();
    >>>       return retval;
    >>> -fail_nomem_vmi_store:
    >>> -    unlink_anon_vmas(tmp);
    >>>   fail_nomem_anon_vma_fork:
    >>>       mpol_put(vma_policy(tmp));
    >>>   fail_nomem_policy:
    >>> diff --git a/mm/mmap.c b/mm/mmap.c
    >>> index b56a7f0c9f85..dfc6881be81c 100644
    >>> --- a/mm/mmap.c
    >>> +++ b/mm/mmap.c
    >>> @@ -3196,7 +3196,11 @@ void exit_mmap(struct mm_struct *mm)
    >>>       arch_exit_mmap(mm);
    >>>       vma = mas_find(&mas, ULONG_MAX);
    >>> -    if (!vma) {
    >>> +    /*
    >>> +     * If dup_mmap() fails to remove a VMA marked VM_DONTCOPY,
    >>> +     * xa_is_zero(vma) may be true.
    >>> +     */
    >>> +    if (!vma || xa_is_zero(vma)) {
    >>>           /* Can happen if dup_mmap() received an OOM */
    >>>           mmap_read_unlock(mm);
    >>>           return;
    >>> @@ -3234,7 +3238,13 @@ void exit_mmap(struct mm_struct *mm)
    >>>           remove_vma(vma, true);
    >>>           count++;
    >>>           cond_resched();
    >>> -    } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL);
    >>> +        vma = mas_find(&mas, ULONG_MAX);
    >>> +        /*
    >>> +         * If xa_is_zero(vma) is true, it means that subsequent VMAs
    >>> +         * donot need to be removed. Can happen if dup_mmap() fails to
    >>> +         * remove a VMA marked VM_DONTCOPY.
    >>> +         */
    >>> +    } while (vma != NULL && !xa_is_zero(vma));
    >>>       BUG_ON(count != mm->map_count);
    >>> --
    >>> 2.20.1
    >>>
    >>

    \
     
     \ /
      Last update: 2023-09-15 12:58    [W:5.412 / U:0.024 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site