lkml.org 
[lkml]   [2008]   [Jul]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: current linux-2.6.git: cpusets completely broken
    Dmitry Adamushko wrote:
    > 2008/7/12 Linus Torvalds <torvalds@linux-foundation.org>:
    >>
    >> On Sat, 12 Jul 2008, Vegard Nossum wrote:
    >>> Can somebody else please test/ack/review it too? This should eventually
    >>> go into 2.6.26 if it doesn't break anything else.
    >> And Dmitry, _please_ also explain what was going on. Why did things break
    >> from calling common_cpu_mem_hotplug_unplug() too much? That function is
    >> called pretty randomly anyway (for just about any random CPU event), so
    >> why did it fail in some circumstances?
    >
    > Upon CPU_DOWN_PREPARE, update_sched_domains() ->
    > detach_destroy_domains(&cpu_online_map) ;
    > does the following:
    >
    > /*
    > * Force a reinitialization of the sched domains hierarchy. The domains
    > * and groups cannot be updated in place without racing with the balancing
    > * code, so we temporarily attach all running cpus to the NULL domain
    > * which will prevent rebalancing while the sched domains are recalculated.
    > */
    >
    > The sched-domains should be rebuilt when a CPU_DOWN ops. is completed,
    > effectivelly either upon CPU_DEAD{_FROZEN} (upon success) or
    > CPU_DOWN_FAILED{_FROZEN} (upon failure -- restore the things to their
    > initial state). That's what update_sched_domains() also does but only
    > for !CPUSETS case.
    >
    > With Max's patch, sched-domains' reinitialization is delegated to CPUSETS code:
    >
    > cpuset_handle_cpuhp() -> common_cpu_mem_hotplug_unplug() ->
    > rebuild_sched_domains()
    >
    > which as you've said "called pretty randomly anyway", e.g. for CPU_UP_PREPARE.
    >
    > [ ah, then rebuild_sched_domains() should not be there. It should be
    > nop for MEMPLUG events I presume - should make another patch. ]
    >
    > Being called for CPU_UP_PREPARE (and if its callback is called after
    > update_sched_domains()), it just negates all the work done by
    > update_sched_domains() -- i.e. a soon-to-be-offline cpu is included in
    > the sched-domains and that makes it visible for the load-balancer
    > while the CPU_DOWN ops. is in progress.
    >
    > __migrate_live_tasks() moves the tasks off a 'dead' cpu (it's already
    > "offline" when this function is called).
    >
    > try_to_wake_up() is called for one of these tasks from another CPU ->
    > the load-balancer (wake_idle()) picks up a "dead" CPU and places the
    > task on it. Then e.g. BUG_ON(rq->nr_running) detects this a bit later
    > -> oops.
    Ah, makes sense. Thanx for the explanation.

    > Now another funny thing is that we probably have a memory leak with
    > common_cpu_mem_hotplug_unplug() "randomly" calling
    > rebuild_sched_domains() and sometimes re-allocating domains when they
    > already exist.
    I beleive that part is ok. We used to have a leak in the scheduler code where
    arch_init_sched_domains() just allocated new masks without freeing the old
    ones. I fixed that. rebuild_sched_domains() -> partition_sched_domains() is
    clean (I think). partition_sched_domains() first does the cleanup and then
    takes ownership of the domain masks.

    btw It's perfectly ok (or at least it has be ok) to call
    rebuild_sched_domains() randomly because it's need to run every time
    sched_load_balance flags in the cpuset change, and on any other even that
    affects domains. As Paul J explained currently that's the only sane way to
    reconstruct the domains based on the cpuset settings.

    Max


    \
     
     \ /
      Last update: 2008-07-12 21:21    [W:2.546 / U:0.040 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site