lkml.org 
[lkml]   [2010]   [Nov]   [3]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    Subject[PATCH 30 of 66] transparent hugepage core
    Date
    From
    From: Andrea Arcangeli <aarcange@redhat.com>

    Lately I've been working to make KVM use hugepages transparently
    without the usual restrictions of hugetlbfs. Some of the restrictions
    I'd like to see removed:

    1) hugepages have to be swappable or the guest physical memory remains
    locked in RAM and can't be paged out to swap

    2) if a hugepage allocation fails, regular pages should be allocated
    instead and mixed in the same vma without any failure and without
    userland noticing

    3) if some task quits and more hugepages become available in the
    buddy, guest physical memory backed by regular pages should be
    relocated on hugepages automatically in regions under
    madvise(MADV_HUGEPAGE) (ideally event driven by waking up the
    kernel deamon if the order=HPAGE_PMD_SHIFT-PAGE_SHIFT list becomes
    not null)

    4) avoidance of reservation and maximization of use of hugepages whenever
    possible. Reservation (needed to avoid runtime fatal faliures) may be ok for
    1 machine with 1 database with 1 database cache with 1 database cache size
    known at boot time. It's definitely not feasible with a virtualization
    hypervisor usage like RHEV-H that runs an unknown number of virtual machines
    with an unknown size of each virtual machine with an unknown amount of
    pagecache that could be potentially useful in the host for guest not using
    O_DIRECT (aka cache=off).

    hugepages in the virtualization hypervisor (and also in the guest!) are
    much more important than in a regular host not using virtualization, becasue
    with NPT/EPT they decrease the tlb-miss cacheline accesses from 24 to 19 in
    case only the hypervisor uses transparent hugepages, and they decrease the
    tlb-miss cacheline accesses from 19 to 15 in case both the linux hypervisor and
    the linux guest both uses this patch (though the guest will limit the addition
    speedup to anonymous regions only for now...). Even more important is that the
    tlb miss handler is much slower on a NPT/EPT guest than for a regular shadow
    paging or no-virtualization scenario. So maximizing the amount of virtual
    memory cached by the TLB pays off significantly more with NPT/EPT than without
    (even if there would be no significant speedup in the tlb-miss runtime).

    The first (and more tedious) part of this work requires allowing the VM to
    handle anonymous hugepages mixed with regular pages transparently on regular
    anonymous vmas. This is what this patch tries to achieve in the least intrusive
    possible way. We want hugepages and hugetlb to be used in a way so that all
    applications can benefit without changes (as usual we leverage the KVM
    virtualization design: by improving the Linux VM at large, KVM gets the
    performance boost too).

    The most important design choice is: always fallback to 4k allocation
    if the hugepage allocation fails! This is the _very_ opposite of some
    large pagecache patches that failed with -EIO back then if a 64k (or
    similar) allocation failed...

    Second important decision (to reduce the impact of the feature on the
    existing pagetable handling code) is that at any time we can split an
    hugepage into 512 regular pages and it has to be done with an
    operation that can't fail. This way the reliability of the swapping
    isn't decreased (no need to allocate memory when we are short on
    memory to swap) and it's trivial to plug a split_huge_page* one-liner
    where needed without polluting the VM. Over time we can teach
    mprotect, mremap and friends to handle pmd_trans_huge natively without
    calling split_huge_page*. The fact it can't fail isn't just for swap:
    if split_huge_page would return -ENOMEM (instead of the current void)
    we'd need to rollback the mprotect from the middle of it (ideally
    including undoing the split_vma) which would be a big change and in
    the very wrong direction (it'd likely be simpler not to call
    split_huge_page at all and to teach mprotect and friends to handle
    hugepages instead of rolling them back from the middle). In short the
    very value of split_huge_page is that it can't fail.

    The collapsing and madvise(MADV_HUGEPAGE) part will remain separated
    and incremental and it'll just be an "harmless" addition later if this
    initial part is agreed upon. It also should be noted that locking-wise
    replacing regular pages with hugepages is going to be very easy if
    compared to what I'm doing below in split_huge_page, as it will only
    happen when page_count(page) matches page_mapcount(page) if we can
    take the PG_lock and mmap_sem in write mode. collapse_huge_page will
    be a "best effort" that (unlike split_huge_page) can fail at the
    minimal sign of trouble and we can try again later. collapse_huge_page
    will be similar to how KSM works and the madvise(MADV_HUGEPAGE) will
    work similar to madvise(MADV_MERGEABLE).

    The default I like is that transparent hugepages are used at page fault time.
    This can be changed with /sys/kernel/mm/transparent_hugepage/enabled. The
    control knob can be set to three values "always", "madvise", "never" which
    mean respectively that hugepages are always used, or only inside
    madvise(MADV_HUGEPAGE) regions, or never used.
    /sys/kernel/mm/transparent_hugepage/defrag instead controls if the hugepage
    allocation should defrag memory aggressively "always", only inside "madvise"
    regions, or "never".

    The pmd_trans_splitting/pmd_trans_huge locking is very solid. The
    put_page (from get_user_page users that can't use mmu notifier like
    O_DIRECT) that runs against a __split_huge_page_refcount instead was a
    pain to serialize in a way that would result always in a coherent page
    count for both tail and head. I think my locking solution with a
    compound_lock taken only after the page_first is valid and is still a
    PageHead should be safe but it surely needs review from SMP race point
    of view. In short there is no current existing way to serialize the
    O_DIRECT final put_page against split_huge_page_refcount so I had to
    invent a new one (O_DIRECT loses knowledge on the mapping status by
    the time gup_fast returns so...). And I didn't want to impact all
    gup/gup_fast users for now, maybe if we change the gup interface
    substantially we can avoid this locking, I admit I didn't think too
    much about it because changing the gup unpinning interface would be
    invasive.

    If we ignored O_DIRECT we could stick to the existing compound
    refcounting code, by simply adding a
    get_user_pages_fast_flags(foll_flags) where KVM (and any other mmu
    notifier user) would call it without FOLL_GET (and if FOLL_GET isn't
    set we'd just BUG_ON if nobody registered itself in the current task
    mmu notifier list yet). But O_DIRECT is fundamental for decent
    performance of virtualized I/O on fast storage so we can't avoid it to
    solve the race of put_page against split_huge_page_refcount to achieve
    a complete hugepage feature for KVM.

    Swap and oom works fine (well just like with regular pages ;). MMU
    notifier is handled transparently too, with the exception of the young
    bit on the pmd, that didn't have a range check but I think KVM will be
    fine because the whole point of hugepages is that EPT/NPT will also
    use a huge pmd when they notice gup returns pages with PageCompound set,
    so they won't care of a range and there's just the pmd young bit to
    check in that case.

    NOTE: in some cases if the L2 cache is small, this may slowdown and
    waste memory during COWs because 4M of memory are accessed in a single
    fault instead of 8k (the payoff is that after COW the program can run
    faster). So we might want to switch the copy_huge_page (and
    clear_huge_page too) to not temporal stores. I also extensively
    researched ways to avoid this cache trashing with a full prefault
    logic that would cow in 8k/16k/32k/64k up to 1M (I can send those
    patches that fully implemented prefault) but I concluded they're not
    worth it and they add an huge additional complexity and they remove all tlb
    benefits until the full hugepage has been faulted in, to save a little bit of
    memory and some cache during app startup, but they still don't improve
    substantially the cache-trashing during startup if the prefault happens in >4k
    chunks. One reason is that those 4k pte entries copied are still mapped on a
    perfectly cache-colored hugepage, so the trashing is the worst one can generate
    in those copies (cow of 4k page copies aren't so well colored so they trashes
    less, but again this results in software running faster after the page fault).
    Those prefault patches allowed things like a pte where post-cow pages were
    local 4k regular anon pages and the not-yet-cowed pte entries were pointing in
    the middle of some hugepage mapped read-only. If it doesn't payoff
    substantially with todays hardware it will payoff even less in the future with
    larger l2 caches, and the prefault logic would blot the VM a lot. If one is
    emebdded transparent_hugepage can be disabled during boot with sysfs or with
    the boot commandline parameter transparent_hugepage=0 (or
    transparent_hugepage=2 to restrict hugepages inside madvise regions) that will
    ensure not a single hugepage is allocated at boot time. It is simple enough to
    just disable transparent hugepage globally and let transparent hugepages be
    allocated selectively by applications in the MADV_HUGEPAGE region (both at page
    fault time, and if enabled with the collapse_huge_page too through the kernel
    daemon).

    This patch supports only hugepages mapped in the pmd, archs that have
    smaller hugepages will not fit in this patch alone. Also some archs like power
    have certain tlb limits that prevents mixing different page size in the same
    regions so they will not fit in this framework that requires "graceful
    fallback" to basic PAGE_SIZE in case of physical memory fragmentation.
    hugetlbfs remains a perfect fit for those because its software limits happen to
    match the hardware limits. hugetlbfs also remains a perfect fit for hugepage
    sizes like 1GByte that cannot be hoped to be found not fragmented after a
    certain system uptime and that would be very expensive to defragment with
    relocation, so requiring reservation. hugetlbfs is the "reservation way", the
    point of transparent hugepages is not to have any reservation at all and
    maximizing the use of cache and hugepages at all times automatically.

    Some performance result:

    vmx andrea # LD_PRELOAD=/usr/lib64/libhugetlbfs.so HUGETLB_MORECORE=yes HUGETLB_PATH=/mnt/huge/ ./largep
    ages3
    memset page fault 1566023
    memset tlb miss 453854
    memset second tlb miss 453321
    random access tlb miss 41635
    random access second tlb miss 41658
    vmx andrea # LD_PRELOAD=/usr/lib64/libhugetlbfs.so HUGETLB_MORECORE=yes HUGETLB_PATH=/mnt/huge/ ./largepages3
    memset page fault 1566471
    memset tlb miss 453375
    memset second tlb miss 453320
    random access tlb miss 41636
    random access second tlb miss 41637
    vmx andrea # ./largepages3
    memset page fault 1566642
    memset tlb miss 453417
    memset second tlb miss 453313
    random access tlb miss 41630
    random access second tlb miss 41647
    vmx andrea # ./largepages3
    memset page fault 1566872
    memset tlb miss 453418
    memset second tlb miss 453315
    random access tlb miss 41618
    random access second tlb miss 41659
    vmx andrea # echo 0 > /proc/sys/vm/transparent_hugepage
    vmx andrea # ./largepages3
    memset page fault 2182476
    memset tlb miss 460305
    memset second tlb miss 460179
    random access tlb miss 44483
    random access second tlb miss 44186
    vmx andrea # ./largepages3
    memset page fault 2182791
    memset tlb miss 460742
    memset second tlb miss 459962
    random access tlb miss 43981
    random access second tlb miss 43988

    ============
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <sys/time.h>

    #define SIZE (3UL*1024*1024*1024)

    int main()
    {
    char *p = malloc(SIZE), *p2;
    struct timeval before, after;

    gettimeofday(&before, NULL);
    memset(p, 0, SIZE);
    gettimeofday(&after, NULL);
    printf("memset page fault %Lu\n",
    (after.tv_sec-before.tv_sec)*1000000UL +
    after.tv_usec-before.tv_usec);

    gettimeofday(&before, NULL);
    memset(p, 0, SIZE);
    gettimeofday(&after, NULL);
    printf("memset tlb miss %Lu\n",
    (after.tv_sec-before.tv_sec)*1000000UL +
    after.tv_usec-before.tv_usec);

    gettimeofday(&before, NULL);
    memset(p, 0, SIZE);
    gettimeofday(&after, NULL);
    printf("memset second tlb miss %Lu\n",
    (after.tv_sec-before.tv_sec)*1000000UL +
    after.tv_usec-before.tv_usec);

    gettimeofday(&before, NULL);
    for (p2 = p; p2 < p+SIZE; p2 += 4096)
    *p2 = 0;
    gettimeofday(&after, NULL);
    printf("random access tlb miss %Lu\n",
    (after.tv_sec-before.tv_sec)*1000000UL +
    after.tv_usec-before.tv_usec);

    gettimeofday(&before, NULL);
    for (p2 = p; p2 < p+SIZE; p2 += 4096)
    *p2 = 0;
    gettimeofday(&after, NULL);
    printf("random access second tlb miss %Lu\n",
    (after.tv_sec-before.tv_sec)*1000000UL +
    after.tv_usec-before.tv_usec);

    return 0;
    }
    ============

    Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
    Acked-by: Rik van Riel <riel@redhat.com>
    Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
    ---
    * * *
    adapt to mm_counter in -mm

    From: Andrea Arcangeli <aarcange@redhat.com>

    The interface changed slightly.

    Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
    Acked-by: Rik van Riel <riel@redhat.com>
    ---
    * * *
    transparent hugepage bootparam

    From: Andrea Arcangeli <aarcange@redhat.com>

    Allow transparent_hugepage=always|never|madvise at boot.

    Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
    ---

    diff --git a/arch/x86/include/asm/pgtable_64.h b/arch/x86/include/asm/pgtable_64.h
    --- a/arch/x86/include/asm/pgtable_64.h
    +++ b/arch/x86/include/asm/pgtable_64.h
    @@ -286,6 +286,11 @@ static inline pmd_t pmd_mkwrite(pmd_t pm
    return pmd_set_flags(pmd, _PAGE_RW);
    }

    +static inline pmd_t pmd_mknotpresent(pmd_t pmd)
    +{
    + return pmd_clear_flags(pmd, _PAGE_PRESENT);
    +}
    +
    #endif /* !__ASSEMBLY__ */

    #endif /* _ASM_X86_PGTABLE_64_H */
    diff --git a/include/linux/gfp.h b/include/linux/gfp.h
    --- a/include/linux/gfp.h
    +++ b/include/linux/gfp.h
    @@ -108,6 +108,9 @@ struct vm_area_struct;
    __GFP_HARDWALL | __GFP_HIGHMEM | \
    __GFP_MOVABLE)
    #define GFP_IOFS (__GFP_IO | __GFP_FS)
    +#define GFP_TRANSHUGE (GFP_HIGHUSER_MOVABLE | __GFP_COMP | \
    + __GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN | \
    + __GFP_NO_KSWAPD)

    #ifdef CONFIG_NUMA
    #define GFP_THISNODE (__GFP_THISNODE | __GFP_NOWARN | __GFP_NORETRY)
    diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
    new file mode 100644
    --- /dev/null
    +++ b/include/linux/huge_mm.h
    @@ -0,0 +1,126 @@
    +#ifndef _LINUX_HUGE_MM_H
    +#define _LINUX_HUGE_MM_H
    +
    +extern int do_huge_pmd_anonymous_page(struct mm_struct *mm,
    + struct vm_area_struct *vma,
    + unsigned long address, pmd_t *pmd,
    + unsigned int flags);
    +extern int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
    + pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
    + struct vm_area_struct *vma);
    +extern int do_huge_pmd_wp_page(struct mm_struct *mm, struct vm_area_struct *vma,
    + unsigned long address, pmd_t *pmd,
    + pmd_t orig_pmd);
    +extern pgtable_t get_pmd_huge_pte(struct mm_struct *mm);
    +extern struct page *follow_trans_huge_pmd(struct mm_struct *mm,
    + unsigned long addr,
    + pmd_t *pmd,
    + unsigned int flags);
    +extern int zap_huge_pmd(struct mmu_gather *tlb,
    + struct vm_area_struct *vma,
    + pmd_t *pmd);
    +
    +enum transparent_hugepage_flag {
    + TRANSPARENT_HUGEPAGE_FLAG,
    + TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG,
    + TRANSPARENT_HUGEPAGE_DEFRAG_FLAG,
    + TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG,
    +#ifdef CONFIG_DEBUG_VM
    + TRANSPARENT_HUGEPAGE_DEBUG_COW_FLAG,
    +#endif
    +};
    +
    +enum page_check_address_pmd_flag {
    + PAGE_CHECK_ADDRESS_PMD_FLAG,
    + PAGE_CHECK_ADDRESS_PMD_NOTSPLITTING_FLAG,
    + PAGE_CHECK_ADDRESS_PMD_SPLITTING_FLAG,
    +};
    +extern pmd_t *page_check_address_pmd(struct page *page,
    + struct mm_struct *mm,
    + unsigned long address,
    + enum page_check_address_pmd_flag flag);
    +
    +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
    +#define HPAGE_PMD_SHIFT HPAGE_SHIFT
    +#define HPAGE_PMD_MASK HPAGE_MASK
    +#define HPAGE_PMD_SIZE HPAGE_SIZE
    +
    +#define transparent_hugepage_enabled(__vma) \
    + (transparent_hugepage_flags & (1<<TRANSPARENT_HUGEPAGE_FLAG) || \
    + (transparent_hugepage_flags & \
    + (1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG) && \
    + (__vma)->vm_flags & VM_HUGEPAGE))
    +#define transparent_hugepage_defrag(__vma) \
    + ((transparent_hugepage_flags & \
    + (1<<TRANSPARENT_HUGEPAGE_DEFRAG_FLAG)) || \
    + (transparent_hugepage_flags & \
    + (1<<TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG) && \
    + (__vma)->vm_flags & VM_HUGEPAGE))
    +#ifdef CONFIG_DEBUG_VM
    +#define transparent_hugepage_debug_cow() \
    + (transparent_hugepage_flags & \
    + (1<<TRANSPARENT_HUGEPAGE_DEBUG_COW_FLAG))
    +#else /* CONFIG_DEBUG_VM */
    +#define transparent_hugepage_debug_cow() 0
    +#endif /* CONFIG_DEBUG_VM */
    +
    +extern unsigned long transparent_hugepage_flags;
    +extern int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
    + pmd_t *dst_pmd, pmd_t *src_pmd,
    + struct vm_area_struct *vma,
    + unsigned long addr, unsigned long end);
    +extern int handle_pte_fault(struct mm_struct *mm,
    + struct vm_area_struct *vma, unsigned long address,
    + pte_t *pte, pmd_t *pmd, unsigned int flags);
    +extern int split_huge_page(struct page *page);
    +extern void __split_huge_page_pmd(struct mm_struct *mm, pmd_t *pmd);
    +#define split_huge_page_pmd(__mm, __pmd) \
    + do { \
    + pmd_t *____pmd = (__pmd); \
    + if (unlikely(pmd_trans_huge(*____pmd))) \
    + __split_huge_page_pmd(__mm, ____pmd); \
    + } while (0)
    +#define wait_split_huge_page(__anon_vma, __pmd) \
    + do { \
    + pmd_t *____pmd = (__pmd); \
    + spin_unlock_wait(&(__anon_vma)->root->lock); \
    + /* \
    + * spin_unlock_wait() is just a loop in C and so the \
    + * CPU can reorder anything around it. \
    + */ \
    + smp_mb(); \
    + BUG_ON(pmd_trans_splitting(*____pmd) || \
    + pmd_trans_huge(*____pmd)); \
    + } while (0)
    +#define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT)
    +#define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER)
    +#if HPAGE_PMD_ORDER > MAX_ORDER
    +#error "hugepages can't be allocated by the buddy allocator"
    +#endif
    +
    +extern unsigned long vma_address(struct page *page, struct vm_area_struct *vma);
    +static inline int PageTransHuge(struct page *page)
    +{
    + VM_BUG_ON(PageTail(page));
    + return PageHead(page);
    +}
    +#else /* CONFIG_TRANSPARENT_HUGEPAGE */
    +#define HPAGE_PMD_SHIFT ({ BUG(); 0; })
    +#define HPAGE_PMD_MASK ({ BUG(); 0; })
    +#define HPAGE_PMD_SIZE ({ BUG(); 0; })
    +
    +#define transparent_hugepage_enabled(__vma) 0
    +
    +#define transparent_hugepage_flags 0UL
    +static inline int split_huge_page(struct page *page)
    +{
    + return 0;
    +}
    +#define split_huge_page_pmd(__mm, __pmd) \
    + do { } while (0)
    +#define wait_split_huge_page(__anon_vma, __pmd) \
    + do { } while (0)
    +#define PageTransHuge(page) 0
    +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
    +
    +#endif /* _LINUX_HUGE_MM_H */
    diff --git a/include/linux/mm.h b/include/linux/mm.h
    --- a/include/linux/mm.h
    +++ b/include/linux/mm.h
    @@ -111,6 +111,9 @@ extern unsigned int kobjsize(const void
    #define VM_SAO 0x20000000 /* Strong Access Ordering (powerpc) */
    #define VM_PFN_AT_MMAP 0x40000000 /* PFNMAP vma that is fully mapped at mmap time */
    #define VM_MERGEABLE 0x80000000 /* KSM may merge identical pages */
    +#if BITS_PER_LONG > 32
    +#define VM_HUGEPAGE 0x100000000UL /* MADV_HUGEPAGE marked this vma */
    +#endif

    #ifndef VM_STACK_DEFAULT_FLAGS /* arch can override this */
    #define VM_STACK_DEFAULT_FLAGS VM_DATA_DEFAULT_FLAGS
    @@ -240,6 +243,7 @@ struct inode;
    * files which need it (119 of them)
    */
    #include <linux/page-flags.h>
    +#include <linux/huge_mm.h>

    /*
    * Methods to modify the page usage count.
    diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
    --- a/include/linux/mm_inline.h
    +++ b/include/linux/mm_inline.h
    @@ -20,11 +20,18 @@ static inline int page_is_file_cache(str
    }

    static inline void
    +__add_page_to_lru_list(struct zone *zone, struct page *page, enum lru_list l,
    + struct list_head *head)
    +{
    + list_add(&page->lru, head);
    + __inc_zone_state(zone, NR_LRU_BASE + l);
    + mem_cgroup_add_lru_list(page, l);
    +}
    +
    +static inline void
    add_page_to_lru_list(struct zone *zone, struct page *page, enum lru_list l)
    {
    - list_add(&page->lru, &zone->lru[l].list);
    - __inc_zone_state(zone, NR_LRU_BASE + l);
    - mem_cgroup_add_lru_list(page, l);
    + __add_page_to_lru_list(zone, page, l, &zone->lru[l].list);
    }

    static inline void
    diff --git a/include/linux/swap.h b/include/linux/swap.h
    --- a/include/linux/swap.h
    +++ b/include/linux/swap.h
    @@ -208,6 +208,8 @@ extern unsigned int nr_free_pagecache_pa
    /* linux/mm/swap.c */
    extern void __lru_cache_add(struct page *, enum lru_list lru);
    extern void lru_cache_add_lru(struct page *, enum lru_list lru);
    +extern void lru_add_page_tail(struct zone* zone,
    + struct page *page, struct page *page_tail);
    extern void activate_page(struct page *);
    extern void mark_page_accessed(struct page *);
    extern void lru_add_drain(void);
    diff --git a/mm/Makefile b/mm/Makefile
    --- a/mm/Makefile
    +++ b/mm/Makefile
    @@ -42,3 +42,4 @@ obj-$(CONFIG_MEMORY_FAILURE) += memory-f
    obj-$(CONFIG_HWPOISON_INJECT) += hwpoison-inject.o
    obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o
    obj-$(CONFIG_DEBUG_KMEMLEAK_TEST) += kmemleak-test.o
    +obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o
    diff --git a/mm/huge_memory.c b/mm/huge_memory.c
    new file mode 100644
    --- /dev/null
    +++ b/mm/huge_memory.c
    @@ -0,0 +1,899 @@
    +/*
    + * Copyright (C) 2009 Red Hat, Inc.
    + *
    + * This work is licensed under the terms of the GNU GPL, version 2. See
    + * the COPYING file in the top-level directory.
    + */
    +
    +#include <linux/mm.h>
    +#include <linux/sched.h>
    +#include <linux/highmem.h>
    +#include <linux/hugetlb.h>
    +#include <linux/mmu_notifier.h>
    +#include <linux/rmap.h>
    +#include <linux/swap.h>
    +#include <asm/tlb.h>
    +#include <asm/pgalloc.h>
    +#include "internal.h"
    +
    +unsigned long transparent_hugepage_flags __read_mostly =
    + (1<<TRANSPARENT_HUGEPAGE_FLAG);
    +
    +#ifdef CONFIG_SYSFS
    +static ssize_t double_flag_show(struct kobject *kobj,
    + struct kobj_attribute *attr, char *buf,
    + enum transparent_hugepage_flag enabled,
    + enum transparent_hugepage_flag req_madv)
    +{
    + if (test_bit(enabled, &transparent_hugepage_flags)) {
    + VM_BUG_ON(test_bit(req_madv, &transparent_hugepage_flags));
    + return sprintf(buf, "[always] madvise never\n");
    + } else if (test_bit(req_madv, &transparent_hugepage_flags))
    + return sprintf(buf, "always [madvise] never\n");
    + else
    + return sprintf(buf, "always madvise [never]\n");
    +}
    +static ssize_t double_flag_store(struct kobject *kobj,
    + struct kobj_attribute *attr,
    + const char *buf, size_t count,
    + enum transparent_hugepage_flag enabled,
    + enum transparent_hugepage_flag req_madv)
    +{
    + if (!memcmp("always", buf,
    + min(sizeof("always")-1, count))) {
    + set_bit(enabled, &transparent_hugepage_flags);
    + clear_bit(req_madv, &transparent_hugepage_flags);
    + } else if (!memcmp("madvise", buf,
    + min(sizeof("madvise")-1, count))) {
    + clear_bit(enabled, &transparent_hugepage_flags);
    + set_bit(req_madv, &transparent_hugepage_flags);
    + } else if (!memcmp("never", buf,
    + min(sizeof("never")-1, count))) {
    + clear_bit(enabled, &transparent_hugepage_flags);
    + clear_bit(req_madv, &transparent_hugepage_flags);
    + } else
    + return -EINVAL;
    +
    + return count;
    +}
    +
    +static ssize_t enabled_show(struct kobject *kobj,
    + struct kobj_attribute *attr, char *buf)
    +{
    + return double_flag_show(kobj, attr, buf,
    + TRANSPARENT_HUGEPAGE_FLAG,
    + TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG);
    +}
    +static ssize_t enabled_store(struct kobject *kobj,
    + struct kobj_attribute *attr,
    + const char *buf, size_t count)
    +{
    + return double_flag_store(kobj, attr, buf, count,
    + TRANSPARENT_HUGEPAGE_FLAG,
    + TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG);
    +}
    +static struct kobj_attribute enabled_attr =
    + __ATTR(enabled, 0644, enabled_show, enabled_store);
    +
    +static ssize_t single_flag_show(struct kobject *kobj,
    + struct kobj_attribute *attr, char *buf,
    + enum transparent_hugepage_flag flag)
    +{
    + if (test_bit(flag, &transparent_hugepage_flags))
    + return sprintf(buf, "[yes] no\n");
    + else
    + return sprintf(buf, "yes [no]\n");
    +}
    +static ssize_t single_flag_store(struct kobject *kobj,
    + struct kobj_attribute *attr,
    + const char *buf, size_t count,
    + enum transparent_hugepage_flag flag)
    +{
    + if (!memcmp("yes", buf,
    + min(sizeof("yes")-1, count))) {
    + set_bit(flag, &transparent_hugepage_flags);
    + } else if (!memcmp("no", buf,
    + min(sizeof("no")-1, count))) {
    + clear_bit(flag, &transparent_hugepage_flags);
    + } else
    + return -EINVAL;
    +
    + return count;
    +}
    +
    +/*
    + * Currently defrag only disables __GFP_NOWAIT for allocation. A blind
    + * __GFP_REPEAT is too aggressive, it's never worth swapping tons of
    + * memory just to allocate one more hugepage.
    + */
    +static ssize_t defrag_show(struct kobject *kobj,
    + struct kobj_attribute *attr, char *buf)
    +{
    + return double_flag_show(kobj, attr, buf,
    + TRANSPARENT_HUGEPAGE_DEFRAG_FLAG,
    + TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG);
    +}
    +static ssize_t defrag_store(struct kobject *kobj,
    + struct kobj_attribute *attr,
    + const char *buf, size_t count)
    +{
    + return double_flag_store(kobj, attr, buf, count,
    + TRANSPARENT_HUGEPAGE_DEFRAG_FLAG,
    + TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG);
    +}
    +static struct kobj_attribute defrag_attr =
    + __ATTR(defrag, 0644, defrag_show, defrag_store);
    +
    +#ifdef CONFIG_DEBUG_VM
    +static ssize_t debug_cow_show(struct kobject *kobj,
    + struct kobj_attribute *attr, char *buf)
    +{
    + return single_flag_show(kobj, attr, buf,
    + TRANSPARENT_HUGEPAGE_DEBUG_COW_FLAG);
    +}
    +static ssize_t debug_cow_store(struct kobject *kobj,
    + struct kobj_attribute *attr,
    + const char *buf, size_t count)
    +{
    + return single_flag_store(kobj, attr, buf, count,
    + TRANSPARENT_HUGEPAGE_DEBUG_COW_FLAG);
    +}
    +static struct kobj_attribute debug_cow_attr =
    + __ATTR(debug_cow, 0644, debug_cow_show, debug_cow_store);
    +#endif /* CONFIG_DEBUG_VM */
    +
    +static struct attribute *hugepage_attr[] = {
    + &enabled_attr.attr,
    + &defrag_attr.attr,
    +#ifdef CONFIG_DEBUG_VM
    + &debug_cow_attr.attr,
    +#endif
    + NULL,
    +};
    +
    +static struct attribute_group hugepage_attr_group = {
    + .attrs = hugepage_attr,
    + .name = "transparent_hugepage",
    +};
    +#endif /* CONFIG_SYSFS */
    +
    +static int __init hugepage_init(void)
    +{
    +#ifdef CONFIG_SYSFS
    + int err;
    +
    + err = sysfs_create_group(mm_kobj, &hugepage_attr_group);
    + if (err)
    + printk(KERN_ERR "hugepage: register sysfs failed\n");
    +#endif
    + return 0;
    +}
    +module_init(hugepage_init)
    +
    +static int __init setup_transparent_hugepage(char *str)
    +{
    + int ret = 0;
    + if (!str)
    + goto out;
    + if (!strcmp(str, "always")) {
    + set_bit(TRANSPARENT_HUGEPAGE_FLAG,
    + &transparent_hugepage_flags);
    + clear_bit(TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG,
    + &transparent_hugepage_flags);
    + ret = 1;
    + } else if (!strcmp(str, "madvise")) {
    + clear_bit(TRANSPARENT_HUGEPAGE_FLAG,
    + &transparent_hugepage_flags);
    + set_bit(TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG,
    + &transparent_hugepage_flags);
    + ret = 1;
    + } else if (!strcmp(str, "never")) {
    + clear_bit(TRANSPARENT_HUGEPAGE_FLAG,
    + &transparent_hugepage_flags);
    + clear_bit(TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG,
    + &transparent_hugepage_flags);
    + ret = 1;
    + }
    +out:
    + if (!ret)
    + printk(KERN_WARNING
    + "transparent_hugepage= cannot parse, ignored\n");
    + return ret;
    +}
    +__setup("transparent_hugepage=", setup_transparent_hugepage);
    +
    +static void prepare_pmd_huge_pte(pgtable_t pgtable,
    + struct mm_struct *mm)
    +{
    + VM_BUG_ON(spin_can_lock(&mm->page_table_lock));
    +
    + /* FIFO */
    + if (!mm->pmd_huge_pte)
    + INIT_LIST_HEAD(&pgtable->lru);
    + else
    + list_add(&pgtable->lru, &mm->pmd_huge_pte->lru);
    + mm->pmd_huge_pte = pgtable;
    +}
    +
    +static inline pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
    +{
    + if (likely(vma->vm_flags & VM_WRITE))
    + pmd = pmd_mkwrite(pmd);
    + return pmd;
    +}
    +
    +static int __do_huge_pmd_anonymous_page(struct mm_struct *mm,
    + struct vm_area_struct *vma,
    + unsigned long haddr, pmd_t *pmd,
    + struct page *page)
    +{
    + int ret = 0;
    + pgtable_t pgtable;
    +
    + VM_BUG_ON(!PageCompound(page));
    + pgtable = pte_alloc_one(mm, haddr);
    + if (unlikely(!pgtable)) {
    + put_page(page);
    + return VM_FAULT_OOM;
    + }
    +
    + clear_huge_page(page, haddr, HPAGE_PMD_NR);
    + __SetPageUptodate(page);
    +
    + spin_lock(&mm->page_table_lock);
    + if (unlikely(!pmd_none(*pmd))) {
    + spin_unlock(&mm->page_table_lock);
    + put_page(page);
    + pte_free(mm, pgtable);
    + } else {
    + pmd_t entry;
    + entry = mk_pmd(page, vma->vm_page_prot);
    + entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
    + entry = pmd_mkhuge(entry);
    + /*
    + * The spinlocking to take the lru_lock inside
    + * page_add_new_anon_rmap() acts as a full memory
    + * barrier to be sure clear_huge_page writes become
    + * visible after the set_pmd_at() write.
    + */
    + page_add_new_anon_rmap(page, vma, haddr);
    + set_pmd_at(mm, haddr, pmd, entry);
    + prepare_pmd_huge_pte(pgtable, mm);
    + add_mm_counter(mm, MM_ANONPAGES, HPAGE_PMD_NR);
    + spin_unlock(&mm->page_table_lock);
    + }
    +
    + return ret;
    +}
    +
    +static inline struct page *alloc_hugepage(int defrag)
    +{
    + return alloc_pages(GFP_TRANSHUGE & ~(defrag ? 0 : __GFP_WAIT),
    + HPAGE_PMD_ORDER);
    +}
    +
    +int do_huge_pmd_anonymous_page(struct mm_struct *mm, struct vm_area_struct *vma,
    + unsigned long address, pmd_t *pmd,
    + unsigned int flags)
    +{
    + struct page *page;
    + unsigned long haddr = address & HPAGE_PMD_MASK;
    + pte_t *pte;
    +
    + if (haddr >= vma->vm_start && haddr + HPAGE_PMD_SIZE <= vma->vm_end) {
    + if (unlikely(anon_vma_prepare(vma)))
    + return VM_FAULT_OOM;
    + page = alloc_hugepage(transparent_hugepage_defrag(vma));
    + if (unlikely(!page))
    + goto out;
    +
    + return __do_huge_pmd_anonymous_page(mm, vma, haddr, pmd, page);
    + }
    +out:
    + /*
    + * Use __pte_alloc instead of pte_alloc_map, because we can't
    + * run pte_offset_map on the pmd, if an huge pmd could
    + * materialize from under us from a different thread.
    + */
    + if (unlikely(__pte_alloc(mm, vma, pmd, address)))
    + return VM_FAULT_OOM;
    + /* if an huge pmd materialized from under us just retry later */
    + if (unlikely(pmd_trans_huge(*pmd)))
    + return 0;
    + /*
    + * A regular pmd is established and it can't morph into a huge pmd
    + * from under us anymore at this point because we hold the mmap_sem
    + * read mode and khugepaged takes it in write mode. So now it's
    + * safe to run pte_offset_map().
    + */
    + pte = pte_offset_map(pmd, address);
    + return handle_pte_fault(mm, vma, address, pte, pmd, flags);
    +}
    +
    +int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
    + pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
    + struct vm_area_struct *vma)
    +{
    + struct page *src_page;
    + pmd_t pmd;
    + pgtable_t pgtable;
    + int ret;
    +
    + ret = -ENOMEM;
    + pgtable = pte_alloc_one(dst_mm, addr);
    + if (unlikely(!pgtable))
    + goto out;
    +
    + spin_lock(&dst_mm->page_table_lock);
    + spin_lock_nested(&src_mm->page_table_lock, SINGLE_DEPTH_NESTING);
    +
    + ret = -EAGAIN;
    + pmd = *src_pmd;
    + if (unlikely(!pmd_trans_huge(pmd))) {
    + pte_free(dst_mm, pgtable);
    + goto out_unlock;
    + }
    + if (unlikely(pmd_trans_splitting(pmd))) {
    + /* split huge page running from under us */
    + spin_unlock(&src_mm->page_table_lock);
    + spin_unlock(&dst_mm->page_table_lock);
    + pte_free(dst_mm, pgtable);
    +
    + wait_split_huge_page(vma->anon_vma, src_pmd); /* src_vma */
    + goto out;
    + }
    + src_page = pmd_page(pmd);
    + VM_BUG_ON(!PageHead(src_page));
    + get_page(src_page);
    + page_dup_rmap(src_page);
    + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
    +
    + pmdp_set_wrprotect(src_mm, addr, src_pmd);
    + pmd = pmd_mkold(pmd_wrprotect(pmd));
    + set_pmd_at(dst_mm, addr, dst_pmd, pmd);
    + prepare_pmd_huge_pte(pgtable, dst_mm);
    +
    + ret = 0;
    +out_unlock:
    + spin_unlock(&src_mm->page_table_lock);
    + spin_unlock(&dst_mm->page_table_lock);
    +out:
    + return ret;
    +}
    +
    +/* no "address" argument so destroys page coloring of some arch */
    +pgtable_t get_pmd_huge_pte(struct mm_struct *mm)
    +{
    + pgtable_t pgtable;
    +
    + VM_BUG_ON(spin_can_lock(&mm->page_table_lock));
    +
    + /* FIFO */
    + pgtable = mm->pmd_huge_pte;
    + if (list_empty(&pgtable->lru))
    + mm->pmd_huge_pte = NULL;
    + else {
    + mm->pmd_huge_pte = list_entry(pgtable->lru.next,
    + struct page, lru);
    + list_del(&pgtable->lru);
    + }
    + return pgtable;
    +}
    +
    +static int do_huge_pmd_wp_page_fallback(struct mm_struct *mm,
    + struct vm_area_struct *vma,
    + unsigned long address,
    + pmd_t *pmd, pmd_t orig_pmd,
    + struct page *page,
    + unsigned long haddr)
    +{
    + pgtable_t pgtable;
    + pmd_t _pmd;
    + int ret = 0, i;
    + struct page **pages;
    +
    + pages = kmalloc(sizeof(struct page *) * HPAGE_PMD_NR,
    + GFP_KERNEL);
    + if (unlikely(!pages)) {
    + ret |= VM_FAULT_OOM;
    + goto out;
    + }
    +
    + for (i = 0; i < HPAGE_PMD_NR; i++) {
    + pages[i] = alloc_page_vma(GFP_HIGHUSER_MOVABLE,
    + vma, address);
    + if (unlikely(!pages[i])) {
    + while (--i >= 0)
    + put_page(pages[i]);
    + kfree(pages);
    + ret |= VM_FAULT_OOM;
    + goto out;
    + }
    + }
    +
    + for (i = 0; i < HPAGE_PMD_NR; i++) {
    + copy_user_highpage(pages[i], page + i,
    + haddr + PAGE_SHIFT*i, vma);
    + __SetPageUptodate(pages[i]);
    + cond_resched();
    + }
    +
    + spin_lock(&mm->page_table_lock);
    + if (unlikely(!pmd_same(*pmd, orig_pmd)))
    + goto out_free_pages;
    + VM_BUG_ON(!PageHead(page));
    +
    + pmdp_clear_flush_notify(vma, haddr, pmd);
    + /* leave pmd empty until pte is filled */
    +
    + pgtable = get_pmd_huge_pte(mm);
    + pmd_populate(mm, &_pmd, pgtable);
    +
    + for (i = 0; i < HPAGE_PMD_NR; i++, haddr += PAGE_SIZE) {
    + pte_t *pte, entry;
    + entry = mk_pte(pages[i], vma->vm_page_prot);
    + entry = maybe_mkwrite(pte_mkdirty(entry), vma);
    + page_add_new_anon_rmap(pages[i], vma, haddr);
    + pte = pte_offset_map(&_pmd, haddr);
    + VM_BUG_ON(!pte_none(*pte));
    + set_pte_at(mm, haddr, pte, entry);
    + pte_unmap(pte);
    + }
    + kfree(pages);
    +
    + mm->nr_ptes++;
    + smp_wmb(); /* make pte visible before pmd */
    + pmd_populate(mm, pmd, pgtable);
    + page_remove_rmap(page);
    + spin_unlock(&mm->page_table_lock);
    +
    + ret |= VM_FAULT_WRITE;
    + put_page(page);
    +
    +out:
    + return ret;
    +
    +out_free_pages:
    + spin_unlock(&mm->page_table_lock);
    + for (i = 0; i < HPAGE_PMD_NR; i++)
    + put_page(pages[i]);
    + kfree(pages);
    + goto out;
    +}
    +
    +int do_huge_pmd_wp_page(struct mm_struct *mm, struct vm_area_struct *vma,
    + unsigned long address, pmd_t *pmd, pmd_t orig_pmd)
    +{
    + int ret = 0;
    + struct page *page, *new_page;
    + unsigned long haddr;
    +
    + VM_BUG_ON(!vma->anon_vma);
    + spin_lock(&mm->page_table_lock);
    + if (unlikely(!pmd_same(*pmd, orig_pmd)))
    + goto out_unlock;
    +
    + page = pmd_page(orig_pmd);
    + VM_BUG_ON(!PageCompound(page) || !PageHead(page));
    + haddr = address & HPAGE_PMD_MASK;
    + if (page_mapcount(page) == 1) {
    + pmd_t entry;
    + entry = pmd_mkyoung(orig_pmd);
    + entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
    + if (pmdp_set_access_flags(vma, haddr, pmd, entry, 1))
    + update_mmu_cache(vma, address, entry);
    + ret |= VM_FAULT_WRITE;
    + goto out_unlock;
    + }
    + get_page(page);
    + spin_unlock(&mm->page_table_lock);
    +
    + if (transparent_hugepage_enabled(vma) &&
    + !transparent_hugepage_debug_cow())
    + new_page = alloc_hugepage(transparent_hugepage_defrag(vma));
    + else
    + new_page = NULL;
    +
    + if (unlikely(!new_page)) {
    + ret = do_huge_pmd_wp_page_fallback(mm, vma, address,
    + pmd, orig_pmd, page, haddr);
    + put_page(page);
    + goto out;
    + }
    +
    + copy_user_huge_page(new_page, page, haddr, vma, HPAGE_PMD_NR);
    + __SetPageUptodate(new_page);
    +
    + spin_lock(&mm->page_table_lock);
    + put_page(page);
    + if (unlikely(!pmd_same(*pmd, orig_pmd)))
    + put_page(new_page);
    + else {
    + pmd_t entry;
    + VM_BUG_ON(!PageHead(page));
    + entry = mk_pmd(new_page, vma->vm_page_prot);
    + entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
    + entry = pmd_mkhuge(entry);
    + pmdp_clear_flush_notify(vma, haddr, pmd);
    + page_add_new_anon_rmap(new_page, vma, haddr);
    + set_pmd_at(mm, haddr, pmd, entry);
    + update_mmu_cache(vma, address, entry);
    + page_remove_rmap(page);
    + put_page(page);
    + ret |= VM_FAULT_WRITE;
    + }
    +out_unlock:
    + spin_unlock(&mm->page_table_lock);
    +out:
    + return ret;
    +}
    +
    +struct page *follow_trans_huge_pmd(struct mm_struct *mm,
    + unsigned long addr,
    + pmd_t *pmd,
    + unsigned int flags)
    +{
    + struct page *page = NULL;
    +
    + VM_BUG_ON(spin_can_lock(&mm->page_table_lock));
    +
    + if (flags & FOLL_WRITE && !pmd_write(*pmd))
    + goto out;
    +
    + page = pmd_page(*pmd);
    + VM_BUG_ON(!PageHead(page));
    + if (flags & FOLL_TOUCH) {
    + pmd_t _pmd;
    + /*
    + * We should set the dirty bit only for FOLL_WRITE but
    + * for now the dirty bit in the pmd is meaningless.
    + * And if the dirty bit will become meaningful and
    + * we'll only set it with FOLL_WRITE, an atomic
    + * set_bit will be required on the pmd to set the
    + * young bit, instead of the current set_pmd_at.
    + */
    + _pmd = pmd_mkyoung(pmd_mkdirty(*pmd));
    + set_pmd_at(mm, addr & HPAGE_PMD_MASK, pmd, _pmd);
    + }
    + page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT;
    + VM_BUG_ON(!PageCompound(page));
    + if (flags & FOLL_GET)
    + get_page(page);
    +
    +out:
    + return page;
    +}
    +
    +int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
    + pmd_t *pmd)
    +{
    + int ret = 0;
    +
    + spin_lock(&tlb->mm->page_table_lock);
    + if (likely(pmd_trans_huge(*pmd))) {
    + if (unlikely(pmd_trans_splitting(*pmd))) {
    + spin_unlock(&tlb->mm->page_table_lock);
    + wait_split_huge_page(vma->anon_vma,
    + pmd);
    + } else {
    + struct page *page;
    + pgtable_t pgtable;
    + pgtable = get_pmd_huge_pte(tlb->mm);
    + page = pmd_page(*pmd);
    + pmd_clear(pmd);
    + page_remove_rmap(page);
    + VM_BUG_ON(page_mapcount(page) < 0);
    + add_mm_counter(tlb->mm, MM_ANONPAGES, -HPAGE_PMD_NR);
    + VM_BUG_ON(!PageHead(page));
    + spin_unlock(&tlb->mm->page_table_lock);
    + tlb_remove_page(tlb, page);
    + pte_free(tlb->mm, pgtable);
    + ret = 1;
    + }
    + } else
    + spin_unlock(&tlb->mm->page_table_lock);
    +
    + return ret;
    +}
    +
    +pmd_t *page_check_address_pmd(struct page *page,
    + struct mm_struct *mm,
    + unsigned long address,
    + enum page_check_address_pmd_flag flag)
    +{
    + pgd_t *pgd;
    + pud_t *pud;
    + pmd_t *pmd, *ret = NULL;
    +
    + if (address & ~HPAGE_PMD_MASK)
    + goto out;
    +
    + pgd = pgd_offset(mm, address);
    + if (!pgd_present(*pgd))
    + goto out;
    +
    + pud = pud_offset(pgd, address);
    + if (!pud_present(*pud))
    + goto out;
    +
    + pmd = pmd_offset(pud, address);
    + if (pmd_none(*pmd))
    + goto out;
    + if (pmd_page(*pmd) != page)
    + goto out;
    + VM_BUG_ON(flag == PAGE_CHECK_ADDRESS_PMD_NOTSPLITTING_FLAG &&
    + pmd_trans_splitting(*pmd));
    + if (pmd_trans_huge(*pmd)) {
    + VM_BUG_ON(flag == PAGE_CHECK_ADDRESS_PMD_SPLITTING_FLAG &&
    + !pmd_trans_splitting(*pmd));
    + ret = pmd;
    + }
    +out:
    + return ret;
    +}
    +
    +static int __split_huge_page_splitting(struct page *page,
    + struct vm_area_struct *vma,
    + unsigned long address)
    +{
    + struct mm_struct *mm = vma->vm_mm;
    + pmd_t *pmd;
    + int ret = 0;
    +
    + spin_lock(&mm->page_table_lock);
    + pmd = page_check_address_pmd(page, mm, address,
    + PAGE_CHECK_ADDRESS_PMD_NOTSPLITTING_FLAG);
    + if (pmd) {
    + /*
    + * We can't temporarily set the pmd to null in order
    + * to split it, the pmd must remain marked huge at all
    + * times or the VM won't take the pmd_trans_huge paths
    + * and it won't wait on the anon_vma->root->lock to
    + * serialize against split_huge_page*.
    + */
    + pmdp_splitting_flush_notify(vma, address, pmd);
    + ret = 1;
    + }
    + spin_unlock(&mm->page_table_lock);
    +
    + return ret;
    +}
    +
    +static void __split_huge_page_refcount(struct page *page)
    +{
    + int i;
    + unsigned long head_index = page->index;
    + struct zone *zone = page_zone(page);
    +
    + /* prevent PageLRU to go away from under us, and freeze lru stats */
    + spin_lock_irq(&zone->lru_lock);
    + compound_lock(page);
    +
    + for (i = 1; i < HPAGE_PMD_NR; i++) {
    + struct page *page_tail = page + i;
    +
    + /* tail_page->_count cannot change */
    + atomic_sub(atomic_read(&page_tail->_count), &page->_count);
    + BUG_ON(page_count(page) <= 0);
    + atomic_add(page_mapcount(page) + 1, &page_tail->_count);
    + BUG_ON(atomic_read(&page_tail->_count) <= 0);
    +
    + /* after clearing PageTail the gup refcount can be released */
    + smp_mb();
    +
    + page_tail->flags &= ~PAGE_FLAGS_CHECK_AT_PREP;
    + page_tail->flags |= (page->flags &
    + ((1L << PG_referenced) |
    + (1L << PG_swapbacked) |
    + (1L << PG_mlocked) |
    + (1L << PG_uptodate)));
    + page_tail->flags |= (1L << PG_dirty);
    +
    + /*
    + * 1) clear PageTail before overwriting first_page
    + * 2) clear PageTail before clearing PageHead for VM_BUG_ON
    + */
    + smp_wmb();
    +
    + /*
    + * __split_huge_page_splitting() already set the
    + * splitting bit in all pmd that could map this
    + * hugepage, that will ensure no CPU can alter the
    + * mapcount on the head page. The mapcount is only
    + * accounted in the head page and it has to be
    + * transferred to all tail pages in the below code. So
    + * for this code to be safe, the split the mapcount
    + * can't change. But that doesn't mean userland can't
    + * keep changing and reading the page contents while
    + * we transfer the mapcount, so the pmd splitting
    + * status is achieved setting a reserved bit in the
    + * pmd, not by clearing the present bit.
    + */
    + BUG_ON(page_mapcount(page_tail));
    + page_tail->_mapcount = page->_mapcount;
    +
    + BUG_ON(page_tail->mapping);
    + page_tail->mapping = page->mapping;
    +
    + page_tail->index = ++head_index;
    +
    + BUG_ON(!PageAnon(page_tail));
    + BUG_ON(!PageUptodate(page_tail));
    + BUG_ON(!PageDirty(page_tail));
    + BUG_ON(!PageSwapBacked(page_tail));
    +
    + lru_add_page_tail(zone, page, page_tail);
    + }
    +
    + ClearPageCompound(page);
    + compound_unlock(page);
    + spin_unlock_irq(&zone->lru_lock);
    +
    + for (i = 1; i < HPAGE_PMD_NR; i++) {
    + struct page *page_tail = page + i;
    + BUG_ON(page_count(page_tail) <= 0);
    + /*
    + * Tail pages may be freed if there wasn't any mapping
    + * like if add_to_swap() is running on a lru page that
    + * had its mapping zapped. And freeing these pages
    + * requires taking the lru_lock so we do the put_page
    + * of the tail pages after the split is complete.
    + */
    + put_page(page_tail);
    + }
    +
    + /*
    + * Only the head page (now become a regular page) is required
    + * to be pinned by the caller.
    + */
    + BUG_ON(page_count(page) <= 0);
    +}
    +
    +static int __split_huge_page_map(struct page *page,
    + struct vm_area_struct *vma,
    + unsigned long address)
    +{
    + struct mm_struct *mm = vma->vm_mm;
    + pmd_t *pmd, _pmd;
    + int ret = 0, i;
    + pgtable_t pgtable;
    + unsigned long haddr;
    +
    + spin_lock(&mm->page_table_lock);
    + pmd = page_check_address_pmd(page, mm, address,
    + PAGE_CHECK_ADDRESS_PMD_SPLITTING_FLAG);
    + if (pmd) {
    + pgtable = get_pmd_huge_pte(mm);
    + pmd_populate(mm, &_pmd, pgtable);
    +
    + for (i = 0, haddr = address; i < HPAGE_PMD_NR;
    + i++, haddr += PAGE_SIZE) {
    + pte_t *pte, entry;
    + BUG_ON(PageCompound(page+i));
    + entry = mk_pte(page + i, vma->vm_page_prot);
    + entry = maybe_mkwrite(pte_mkdirty(entry), vma);
    + if (!pmd_write(*pmd))
    + entry = pte_wrprotect(entry);
    + else
    + BUG_ON(page_mapcount(page) != 1);
    + if (!pmd_young(*pmd))
    + entry = pte_mkold(entry);
    + pte = pte_offset_map(&_pmd, haddr);
    + BUG_ON(!pte_none(*pte));
    + set_pte_at(mm, haddr, pte, entry);
    + pte_unmap(pte);
    + }
    +
    + mm->nr_ptes++;
    + smp_wmb(); /* make pte visible before pmd */
    + /*
    + * Up to this point the pmd is present and huge and
    + * userland has the whole access to the hugepage
    + * during the split (which happens in place). If we
    + * overwrite the pmd with the not-huge version
    + * pointing to the pte here (which of course we could
    + * if all CPUs were bug free), userland could trigger
    + * a small page size TLB miss on the small sized TLB
    + * while the hugepage TLB entry is still established
    + * in the huge TLB. Some CPU doesn't like that. See
    + * http://support.amd.com/us/Processor_TechDocs/41322.pdf,
    + * Erratum 383 on page 93. Intel should be safe but is
    + * also warns that it's only safe if the permission
    + * and cache attributes of the two entries loaded in
    + * the two TLB is identical (which should be the case
    + * here). But it is generally safer to never allow
    + * small and huge TLB entries for the same virtual
    + * address to be loaded simultaneously. So instead of
    + * doing "pmd_populate(); flush_tlb_range();" we first
    + * mark the current pmd notpresent (atomically because
    + * here the pmd_trans_huge and pmd_trans_splitting
    + * must remain set at all times on the pmd until the
    + * split is complete for this pmd), then we flush the
    + * SMP TLB and finally we write the non-huge version
    + * of the pmd entry with pmd_populate.
    + */
    + set_pmd_at(mm, address, pmd, pmd_mknotpresent(*pmd));
    + flush_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
    + pmd_populate(mm, pmd, pgtable);
    + ret = 1;
    + }
    + spin_unlock(&mm->page_table_lock);
    +
    + return ret;
    +}
    +
    +/* must be called with anon_vma->root->lock hold */
    +static void __split_huge_page(struct page *page,
    + struct anon_vma *anon_vma)
    +{
    + int mapcount, mapcount2;
    + struct anon_vma_chain *avc;
    +
    + BUG_ON(!PageHead(page));
    + BUG_ON(PageTail(page));
    +
    + mapcount = 0;
    + list_for_each_entry(avc, &anon_vma->head, same_anon_vma) {
    + struct vm_area_struct *vma = avc->vma;
    + unsigned long addr = vma_address(page, vma);
    + if (addr == -EFAULT)
    + continue;
    + mapcount += __split_huge_page_splitting(page, vma, addr);
    + }
    + BUG_ON(mapcount != page_mapcount(page));
    +
    + __split_huge_page_refcount(page);
    +
    + mapcount2 = 0;
    + list_for_each_entry(avc, &anon_vma->head, same_anon_vma) {
    + struct vm_area_struct *vma = avc->vma;
    + unsigned long addr = vma_address(page, vma);
    + if (addr == -EFAULT)
    + continue;
    + mapcount2 += __split_huge_page_map(page, vma, addr);
    + }
    + BUG_ON(mapcount != mapcount2);
    +}
    +
    +int split_huge_page(struct page *page)
    +{
    + struct anon_vma *anon_vma;
    + int ret = 1;
    +
    + BUG_ON(!PageAnon(page));
    + anon_vma = page_lock_anon_vma(page);
    + if (!anon_vma)
    + goto out;
    + ret = 0;
    + if (!PageCompound(page))
    + goto out_unlock;
    +
    + BUG_ON(!PageSwapBacked(page));
    + __split_huge_page(page, anon_vma);
    +
    + BUG_ON(PageCompound(page));
    +out_unlock:
    + page_unlock_anon_vma(anon_vma);
    +out:
    + return ret;
    +}
    +
    +void __split_huge_page_pmd(struct mm_struct *mm, pmd_t *pmd)
    +{
    + struct page *page;
    +
    + spin_lock(&mm->page_table_lock);
    + if (unlikely(!pmd_trans_huge(*pmd))) {
    + spin_unlock(&mm->page_table_lock);
    + return;
    + }
    + page = pmd_page(*pmd);
    + VM_BUG_ON(!page_count(page));
    + get_page(page);
    + spin_unlock(&mm->page_table_lock);
    +
    + split_huge_page(page);
    +
    + put_page(page);
    + BUG_ON(pmd_trans_huge(*pmd));
    +}
    diff --git a/mm/memory.c b/mm/memory.c
    --- a/mm/memory.c
    +++ b/mm/memory.c
    @@ -726,9 +726,9 @@ out_set_pte:
    return 0;
    }

    -static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
    - pmd_t *dst_pmd, pmd_t *src_pmd, struct vm_area_struct *vma,
    - unsigned long addr, unsigned long end)
    +int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
    + pmd_t *dst_pmd, pmd_t *src_pmd, struct vm_area_struct *vma,
    + unsigned long addr, unsigned long end)
    {
    pte_t *orig_src_pte, *orig_dst_pte;
    pte_t *src_pte, *dst_pte;
    @@ -802,6 +802,16 @@ static inline int copy_pmd_range(struct
    src_pmd = pmd_offset(src_pud, addr);
    do {
    next = pmd_addr_end(addr, end);
    + if (pmd_trans_huge(*src_pmd)) {
    + int err;
    + err = copy_huge_pmd(dst_mm, src_mm,
    + dst_pmd, src_pmd, addr, vma);
    + if (err == -ENOMEM)
    + return -ENOMEM;
    + if (!err)
    + continue;
    + /* fall through */
    + }
    if (pmd_none_or_clear_bad(src_pmd))
    continue;
    if (copy_pte_range(dst_mm, src_mm, dst_pmd, src_pmd,
    @@ -1004,6 +1014,15 @@ static inline unsigned long zap_pmd_rang
    pmd = pmd_offset(pud, addr);
    do {
    next = pmd_addr_end(addr, end);
    + if (pmd_trans_huge(*pmd)) {
    + if (next-addr != HPAGE_PMD_SIZE)
    + split_huge_page_pmd(vma->vm_mm, pmd);
    + else if (zap_huge_pmd(tlb, vma, pmd)) {
    + (*zap_work)--;
    + continue;
    + }
    + /* fall through */
    + }
    if (pmd_none_or_clear_bad(pmd)) {
    (*zap_work)--;
    continue;
    @@ -1280,11 +1299,27 @@ struct page *follow_page(struct vm_area_
    pmd = pmd_offset(pud, address);
    if (pmd_none(*pmd))
    goto no_page_table;
    - if (pmd_huge(*pmd)) {
    + if (pmd_huge(*pmd) && vma->vm_flags & VM_HUGETLB) {
    BUG_ON(flags & FOLL_GET);
    page = follow_huge_pmd(mm, address, pmd, flags & FOLL_WRITE);
    goto out;
    }
    + if (pmd_trans_huge(*pmd)) {
    + spin_lock(&mm->page_table_lock);
    + if (likely(pmd_trans_huge(*pmd))) {
    + if (unlikely(pmd_trans_splitting(*pmd))) {
    + spin_unlock(&mm->page_table_lock);
    + wait_split_huge_page(vma->anon_vma, pmd);
    + } else {
    + page = follow_trans_huge_pmd(mm, address,
    + pmd, flags);
    + spin_unlock(&mm->page_table_lock);
    + goto out;
    + }
    + } else
    + spin_unlock(&mm->page_table_lock);
    + /* fall through */
    + }
    if (unlikely(pmd_bad(*pmd)))
    goto no_page_table;

    @@ -3141,9 +3176,9 @@ static int do_nonlinear_fault(struct mm_
    * but allow concurrent faults), and pte mapped but not yet locked.
    * We return with mmap_sem still held, but pte unmapped and unlocked.
    */
    -static inline int handle_pte_fault(struct mm_struct *mm,
    - struct vm_area_struct *vma, unsigned long address,
    - pte_t *pte, pmd_t *pmd, unsigned int flags)
    +int handle_pte_fault(struct mm_struct *mm,
    + struct vm_area_struct *vma, unsigned long address,
    + pte_t *pte, pmd_t *pmd, unsigned int flags)
    {
    pte_t entry;
    spinlock_t *ptl;
    @@ -3222,9 +3257,40 @@ int handle_mm_fault(struct mm_struct *mm
    pmd = pmd_alloc(mm, pud, address);
    if (!pmd)
    return VM_FAULT_OOM;
    - pte = pte_alloc_map(mm, vma, pmd, address);
    - if (!pte)
    + if (pmd_none(*pmd) && transparent_hugepage_enabled(vma)) {
    + if (!vma->vm_ops)
    + return do_huge_pmd_anonymous_page(mm, vma, address,
    + pmd, flags);
    + } else {
    + pmd_t orig_pmd = *pmd;
    + barrier();
    + if (pmd_trans_huge(orig_pmd)) {
    + if (flags & FAULT_FLAG_WRITE &&
    + !pmd_write(orig_pmd) &&
    + !pmd_trans_splitting(orig_pmd))
    + return do_huge_pmd_wp_page(mm, vma, address,
    + pmd, orig_pmd);
    + return 0;
    + }
    + }
    +
    + /*
    + * Use __pte_alloc instead of pte_alloc_map, because we can't
    + * run pte_offset_map on the pmd, if an huge pmd could
    + * materialize from under us from a different thread.
    + */
    + if (unlikely(__pte_alloc(mm, vma, pmd, address)))
    return VM_FAULT_OOM;
    + /* if an huge pmd materialized from under us just retry later */
    + if (unlikely(pmd_trans_huge(*pmd)))
    + return 0;
    + /*
    + * A regular pmd is established and it can't morph into a huge pmd
    + * from under us anymore at this point because we hold the mmap_sem
    + * read mode and khugepaged takes it in write mode. So now it's
    + * safe to run pte_offset_map().
    + */
    + pte = pte_offset_map(pmd, address);

    return handle_pte_fault(mm, vma, address, pte, pmd, flags);
    }
    diff --git a/mm/rmap.c b/mm/rmap.c
    --- a/mm/rmap.c
    +++ b/mm/rmap.c
    @@ -360,7 +360,7 @@ void page_unlock_anon_vma(struct anon_vm
    * Returns virtual address or -EFAULT if page's index/offset is not
    * within the range mapped the @vma.
    */
    -static inline unsigned long
    +inline unsigned long
    vma_address(struct page *page, struct vm_area_struct *vma)
    {
    pgoff_t pgoff = page->index << (PAGE_CACHE_SHIFT - PAGE_SHIFT);
    @@ -435,6 +435,8 @@ pte_t *__page_check_address(struct page
    pmd = pmd_offset(pud, address);
    if (!pmd_present(*pmd))
    return NULL;
    + if (pmd_trans_huge(*pmd))
    + return NULL;

    pte = pte_offset_map(pmd, address);
    /* Make a quick check before getting the lock */
    @@ -489,35 +491,17 @@ int page_referenced_one(struct page *pag
    unsigned long *vm_flags)
    {
    struct mm_struct *mm = vma->vm_mm;
    - pte_t *pte;
    - spinlock_t *ptl;
    int referenced = 0;

    - pte = page_check_address(page, mm, address, &ptl, 0);
    - if (!pte)
    - goto out;
    -
    /*
    * Don't want to elevate referenced for mlocked page that gets this far,
    * in order that it progresses to try_to_unmap and is moved to the
    * unevictable list.
    */
    if (vma->vm_flags & VM_LOCKED) {
    - *mapcount = 1; /* break early from loop */
    + *mapcount = 0; /* break early from loop */
    *vm_flags |= VM_LOCKED;
    - goto out_unmap;
    - }
    -
    - if (ptep_clear_flush_young_notify(vma, address, pte)) {
    - /*
    - * Don't treat a reference through a sequentially read
    - * mapping as such. If the page has been used in
    - * another mapping, we will catch it; if this other
    - * mapping is already gone, the unmap path will have
    - * set PG_referenced or activated the page.
    - */
    - if (likely(!VM_SequentialReadHint(vma)))
    - referenced++;
    + goto out;
    }

    /* Pretend the page is referenced if the task has the
    @@ -526,9 +510,39 @@ int page_referenced_one(struct page *pag
    rwsem_is_locked(&mm->mmap_sem))
    referenced++;

    -out_unmap:
    + if (unlikely(PageTransHuge(page))) {
    + pmd_t *pmd;
    +
    + spin_lock(&mm->page_table_lock);
    + pmd = page_check_address_pmd(page, mm, address,
    + PAGE_CHECK_ADDRESS_PMD_FLAG);
    + if (pmd && !pmd_trans_splitting(*pmd) &&
    + pmdp_clear_flush_young_notify(vma, address, pmd))
    + referenced++;
    + spin_unlock(&mm->page_table_lock);
    + } else {
    + pte_t *pte;
    + spinlock_t *ptl;
    +
    + pte = page_check_address(page, mm, address, &ptl, 0);
    + if (!pte)
    + goto out;
    +
    + if (ptep_clear_flush_young_notify(vma, address, pte)) {
    + /*
    + * Don't treat a reference through a sequentially read
    + * mapping as such. If the page has been used in
    + * another mapping, we will catch it; if this other
    + * mapping is already gone, the unmap path will have
    + * set PG_referenced or activated the page.
    + */
    + if (likely(!VM_SequentialReadHint(vma)))
    + referenced++;
    + }
    + pte_unmap_unlock(pte, ptl);
    + }
    +
    (*mapcount)--;
    - pte_unmap_unlock(pte, ptl);

    if (referenced)
    *vm_flags |= vma->vm_flags;
    diff --git a/mm/swap.c b/mm/swap.c
    --- a/mm/swap.c
    +++ b/mm/swap.c
    @@ -465,6 +465,43 @@ void __pagevec_release(struct pagevec *p

    EXPORT_SYMBOL(__pagevec_release);

    +/* used by __split_huge_page_refcount() */
    +void lru_add_page_tail(struct zone* zone,
    + struct page *page, struct page *page_tail)
    +{
    + int active;
    + enum lru_list lru;
    + const int file = 0;
    + struct list_head *head;
    +
    + VM_BUG_ON(!PageHead(page));
    + VM_BUG_ON(PageCompound(page_tail));
    + VM_BUG_ON(PageLRU(page_tail));
    + VM_BUG_ON(!spin_is_locked(&zone->lru_lock));
    +
    + SetPageLRU(page_tail);
    +
    + if (page_evictable(page_tail, NULL)) {
    + if (PageActive(page)) {
    + SetPageActive(page_tail);
    + active = 1;
    + lru = LRU_ACTIVE_ANON;
    + } else {
    + active = 0;
    + lru = LRU_INACTIVE_ANON;
    + }
    + update_page_reclaim_stat(zone, page_tail, file, active);
    + if (likely(PageLRU(page)))
    + head = page->lru.prev;
    + else
    + head = &zone->lru[lru].list;
    + __add_page_to_lru_list(zone, page_tail, lru, head);
    + } else {
    + SetPageUnevictable(page_tail);
    + add_page_to_lru_list(zone, page_tail, LRU_UNEVICTABLE);
    + }
    +}
    +
    /*
    * Add the passed pages to the LRU, then drop the caller's refcount
    * on them. Reinitialises the caller's pagevec.

    \
     
     \ /
      Last update: 2010-11-03 16:53    [W:3.921 / U:0.032 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site