Messages in this thread | | | Date | Wed, 18 Mar 2015 14:59:25 +0900 | Subject | Re: [PATCHv2] mm/slub: fix lockups on PREEMPT && !SMP kernels | From | Joonsoo Kim <> |
| |
2015-03-17 21:15 GMT+09:00 Mark Rutland <mark.rutland@arm.com>: > Commit 9aabf810a67cd97e ("mm/slub: optimize alloc/free fastpath by > removing preemption on/off") introduced an occasional hang for kernels > built with CONFIG_PREEMPT && !CONFIG_SMP. > > The problem is the following loop the patch introduced to > slab_alloc_node and slab_free: > > do { > tid = this_cpu_read(s->cpu_slab->tid); > c = raw_cpu_ptr(s->cpu_slab); > } while (IS_ENABLED(CONFIG_PREEMPT) && unlikely(tid != c->tid)); > > GCC 4.9 has been observed to hoist the load of c and c->tid above the > loop for !SMP kernels (as in this case raw_cpu_ptr(x) is compile-time > constant and does not force a reload). On arm64 the generated assembly > looks like: > > ffffffc00016d3c4: f9400404 ldr x4, [x0,#8] > ffffffc00016d3c8: f9400401 ldr x1, [x0,#8] > ffffffc00016d3cc: eb04003f cmp x1, x4 > ffffffc00016d3d0: 54ffffc1 b.ne ffffffc00016d3c8 <slab_alloc_node.constprop.82+0x30> > > If the thread is preempted between the load of c->tid (into x1) and tid > (into x4), and an allocation or free occurs in another thread (bumping > the cpu_slab's tid), the thread will be stuck in the loop until > s->cpu_slab->tid wraps, which may be forever in the absence of > allocations/frees on the same CPU. > > This patch changes the loop condition to access c->tid with READ_ONCE. > This ensures that the value is reloaded even when the compiler would > otherwise assume it could cache the value, and also ensures that the > load will not be torn. > > Signed-off-by: Mark Rutland <mark.rutland@arm.com> > Cc: Andrew Morton <akpm@linux-foundation.org> > Cc: Catalin Marinas <catalin.marinas@arm.com> > Cc: Christoph Lameter <cl@linux.com> > Cc: David Rientjes <rientjes@google.com> > Cc: Jesper Dangaard Brouer <brouer@redhat.com> > Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> > Cc: Linus Torvalds <torvalds@linux-foundation.org> > Cc: Pekka Enberg <penberg@kernel.org> > Cc: Steve Capper <steve.capper@linaro.org> > --- > mm/slub.c | 6 ++++-- > 1 file changed, 4 insertions(+), 2 deletions(-) > > Since v1 [1]: > * Do not erroneously remove the loop > > [1] lkml.kernel.org/r/1426261632-8911-1-git-send-email-mark.rutland@arm.com > > diff --git a/mm/slub.c b/mm/slub.c > index 6832c4e..82c4737 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -2449,7 +2449,8 @@ redo: > do { > tid = this_cpu_read(s->cpu_slab->tid); > c = raw_cpu_ptr(s->cpu_slab); > - } while (IS_ENABLED(CONFIG_PREEMPT) && unlikely(tid != c->tid)); > + } while (IS_ENABLED(CONFIG_PREEMPT) && > + unlikely(tid != READ_ONCE(c->tid))); > > /* > * Irqless object alloc/free algorithm used here depends on sequence > @@ -2718,7 +2719,8 @@ redo: > do { > tid = this_cpu_read(s->cpu_slab->tid); > c = raw_cpu_ptr(s->cpu_slab); > - } while (IS_ENABLED(CONFIG_PREEMPT) && unlikely(tid != c->tid)); > + } while (IS_ENABLED(CONFIG_PREEMPT) && > + unlikely(tid != READ_ONCE(c->tid))); > > /* Same with comment on barrier() in slab_alloc_node() */ > barrier(); > --
Hello,
Could you show me generated code again?
What we need to check is redoing whole things in the loop. Previous attached code seems to me that it already did refetching c->tid in the loop and this patch looks only handle refetching c->tid. READ_ONCE(c->tid) will trigger redoing 'tid = this_cpu_read(s->cpu_slab->tid)'?
Thanks.
| |