lkml.org 
[lkml]   [2021]   [Jul]   [29]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    From
    Subject[PATCH v3 11/35] mm, slub: simplify kmem_cache_cpu and tid setup
    Date
    In slab_alloc_node() and do_slab_free() fastpaths we need to guarantee that
    our kmem_cache_cpu pointer is from the same cpu as the tid value. Currently
    that's done by reading the tid first using this_cpu_read(), then the
    kmem_cache_cpu pointer and verifying we read the same tid using the pointer and
    plain READ_ONCE().

    This can be simplified to just fetching kmem_cache_cpu pointer and then reading
    tid using the pointer. That guarantees they are from the same cpu. We don't
    need to read the tid using this_cpu_read() because the value will be validated
    by this_cpu_cmpxchg_double(), making sure we are on the correct cpu and the
    freelist didn't change by anyone preempting us since reading the tid.

    Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
    Acked-by: Mel Gorman <mgorman@techsingularity.net>
    ---
    mm/slub.c | 22 +++++++++-------------
    1 file changed, 9 insertions(+), 13 deletions(-)

    diff --git a/mm/slub.c b/mm/slub.c
    index 469aa8155663..0871bba8ecc2 100644
    --- a/mm/slub.c
    +++ b/mm/slub.c
    @@ -2865,15 +2865,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,
    * reading from one cpu area. That does not matter as long
    * as we end up on the original cpu again when doing the cmpxchg.
    *
    - * We should guarantee that tid and kmem_cache are retrieved on
    - * the same cpu. It could be different if CONFIG_PREEMPTION so we need
    - * to check if it is matched or not.
    + * We must guarantee that tid and kmem_cache_cpu are retrieved on the
    + * same cpu. We read first the kmem_cache_cpu pointer and use it to read
    + * the tid. If we are preempted and switched to another cpu between the
    + * two reads, it's OK as the two are still associated with the same cpu
    + * and cmpxchg later will validate the cpu.
    */
    - do {
    - tid = this_cpu_read(s->cpu_slab->tid);
    - c = raw_cpu_ptr(s->cpu_slab);
    - } while (IS_ENABLED(CONFIG_PREEMPTION) &&
    - unlikely(tid != READ_ONCE(c->tid)));
    + c = raw_cpu_ptr(s->cpu_slab);
    + tid = READ_ONCE(c->tid);

    /*
    * Irqless object alloc/free algorithm used here depends on sequence
    @@ -3147,11 +3146,8 @@ static __always_inline void do_slab_free(struct kmem_cache *s,
    * data is retrieved via this pointer. If we are on the same cpu
    * during the cmpxchg then the free will succeed.
    */
    - do {
    - tid = this_cpu_read(s->cpu_slab->tid);
    - c = raw_cpu_ptr(s->cpu_slab);
    - } while (IS_ENABLED(CONFIG_PREEMPTION) &&
    - unlikely(tid != READ_ONCE(c->tid)));
    + c = raw_cpu_ptr(s->cpu_slab);
    + tid = READ_ONCE(c->tid);

    /* Same with comment on barrier() in slab_alloc_node() */
    barrier();
    --
    2.32.0
    \
     
     \ /
      Last update: 2021-07-29 15:23    [W:4.197 / U:0.932 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site