lkml.org 
[lkml]   [2009]   [Dec]   [22]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: 2.6.33-rc1 unusable due to scheduler issues, circular locking, WARNs and BUGs
From
Date
On Tue, 2009-12-22 at 09:48 +0100, Peter Zijlstra wrote:
> On Mon, 2009-12-21 at 19:17 -0500, Eric Paris wrote:
> > Trying to build a kernel on a 48 core x86_64 box using make -j 64 and
> > I'm exploding in the scheduler. I'm running (and building) kernel
> > f7b84a6ba7eaeba4e1df8feddca1473a7db369a5 There are three distinct
> > signatures of problems. Some boots I'll see all 3 of these failures
> > sometimes only 1 or 2 of them. That's the reason they are kinda split
> > up in dmesg.

Appears the kernel built with no oops, circular locking, or bugs. Only
thing in dmesg during the build was:

DMA-API: debugging out of memory - disabling

I'm going to do it a number more times to be sure, but I had 100%
failure rate before. Sounds like this patch has got to go.

-Eric

> >
> > 1) gcc/3141 is trying to acquire lock:
> > (&(&sem->wait_lock)->rlock){......}, at: [<ffffffff81223234>] __down_read_trylock+0x13/0x46
> >
> > but task is already holding lock:
> > (&rq->lock){-.-.-.}, at: [<ffffffff8103dd2d>] task_rq_lock+0x51/0x83
>
> This is due to the pagefalut happening while holding the rq->lock, so
> its an artefact of 3).
>
> > 2) WARN() in kernel/sched_fair.c:1001 hrtick_start_fair()
>
> Worrying, but probably due to the same problem as 3)
>
> > 3) NULL pointer dereference at 0000000000000168 in check_preempt_wakeup
> > kernel/sched_fair.c
>
> Right, hard to tell where exactly it goes bang, but could you please try
> reverting the below patch.
>
> What I suspect happens is that we his the task_cpu(p)==cpu case, we then
> don't do __set_task_cpu()->set_task_rq(), which sets the group
> scheduling pointers (you seem to have cgroup scheduling enabled).
>
> If those pointers are wild all kinds of interesting bits can happen,
> including 3) and possibly 2).
>
> If this revert doesn't help, could you please also provide the output of
> addr2line -e vmlinux <FAULT_IP> ?
>
> ---
> commit 738d2be4301007f054541c5c4bf7fb6a361c9b3a
> Author: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Date: Wed Dec 16 18:04:42 2009 +0100
>
> sched: Simplify set_task_cpu()
>
> Rearrange code a bit now that its a simpler function.
>
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Cc: Mike Galbraith <efault@gmx.de>
> LKML-Reference: <20091216170518.269101883@chello.nl>
> Signed-off-by: Ingo Molnar <mingo@elte.hu>
>
> diff --git a/kernel/sched.c b/kernel/sched.c
> index f92ce63..8a2bfd3 100644
> --- a/kernel/sched.c
> +++ b/kernel/sched.c
> @@ -2034,11 +2034,8 @@ task_hot(struct task_struct *p, u64 now, struct sched_domain *sd)
> return delta < (s64)sysctl_sched_migration_cost;
> }
>
> -
> void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
> {
> - int old_cpu = task_cpu(p);
> -
> #ifdef CONFIG_SCHED_DEBUG
> /*
> * We should never call set_task_cpu() on a blocked task,
> @@ -2049,11 +2046,11 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
>
> trace_sched_migrate_task(p, new_cpu);
>
> - if (old_cpu != new_cpu) {
> - p->se.nr_migrations++;
> - perf_sw_event(PERF_COUNT_SW_CPU_MIGRATIONS,
> - 1, 1, NULL, 0);
> - }
> + if (task_cpu(p) == new_cpu)
> + return;
> +
> + p->se.nr_migrations++;
> + perf_sw_event(PERF_COUNT_SW_CPU_MIGRATIONS, 1, 1, NULL, 0);
>
> __set_task_cpu(p, new_cpu);
> }
>
>




\
 
 \ /
  Last update: 2009-12-22 15:33    [W:0.097 / U:0.052 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site