lkml.org 
[lkml]   [2008]   [Jul]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: current linux-2.6.git: cpusets completely broken
    2008/7/12 Linus Torvalds <torvalds@linux-foundation.org>:
    >
    >
    > On Sat, 12 Jul 2008, Vegard Nossum wrote:
    >>
    >> Can somebody else please test/ack/review it too? This should eventually
    >> go into 2.6.26 if it doesn't break anything else.
    >
    > And Dmitry, _please_ also explain what was going on. Why did things break
    > from calling common_cpu_mem_hotplug_unplug() too much? That function is
    > called pretty randomly anyway (for just about any random CPU event), so
    > why did it fail in some circumstances?

    Upon CPU_DOWN_PREPARE, update_sched_domains() ->
    detach_destroy_domains(&cpu_online_map) ;
    does the following:

    /*
    * Force a reinitialization of the sched domains hierarchy. The domains
    * and groups cannot be updated in place without racing with the balancing
    * code, so we temporarily attach all running cpus to the NULL domain
    * which will prevent rebalancing while the sched domains are recalculated.
    */

    The sched-domains should be rebuilt when a CPU_DOWN ops. is completed,
    effectivelly either upon CPU_DEAD{_FROZEN} (upon success) or
    CPU_DOWN_FAILED{_FROZEN} (upon failure -- restore the things to their
    initial state). That's what update_sched_domains() also does but only
    for !CPUSETS case.

    With Max's patch, sched-domains' reinitialization is delegated to CPUSETS code:

    cpuset_handle_cpuhp() -> common_cpu_mem_hotplug_unplug() ->
    rebuild_sched_domains()

    which as you've said "called pretty randomly anyway", e.g. for CPU_UP_PREPARE.

    [ ah, then rebuild_sched_domains() should not be there. It should be
    nop for MEMPLUG events I presume - should make another patch. ]

    Being called for CPU_UP_PREPARE (and if its callback is called after
    update_sched_domains()), it just negates all the work done by
    update_sched_domains() -- i.e. a soon-to-be-offline cpu is included in
    the sched-domains and that makes it visible for the load-balancer
    while the CPU_DOWN ops. is in progress.

    __migrate_live_tasks() moves the tasks off a 'dead' cpu (it's already
    "offline" when this function is called).

    try_to_wake_up() is called for one of these tasks from another CPU ->
    the load-balancer (wake_idle()) picks up a "dead" CPU and places the
    task on it. Then e.g. BUG_ON(rq->nr_running) detects this a bit later
    -> oops.

    Now another funny thing is that we probably have a memory leak with
    common_cpu_mem_hotplug_unplug() "randomly" calling
    rebuild_sched_domains() and sometimes re-allocating domains when they
    already exist.


    >
    > Linus
    >

    --
    Best regards,
    Dmitry Adamushko


    \
     
     \ /
      Last update: 2008-07-12 12:07    [W:4.067 / U:0.032 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site