Messages in this thread | | | Date | Thu, 11 Apr 2013 08:48:37 -0500 | From | Robin Holt <> | Subject | Re: Bulk CPU Hotplug (Was Re: [PATCH] Do not force shutdown/reboot to boot cpu.) |
| |
For the v3.9 release, can we consider my awful patch?
Thanks, Robin
On Thu, Apr 11, 2013 at 06:15:18PM +0530, Srivatsa S. Bhat wrote: > On 04/11/2013 11:01 AM, Paul Mackerras wrote: > > On Wed, Apr 10, 2013 at 08:10:05AM -0700, Linus Torvalds wrote: > >> The optimal solution would be to just speed up the > >> disable_nonboot_cpus() code so much that it isn't an issue. That would > >> be good for suspending too, although I guess suspend isn't a big issue > >> if you have a thousand CPU's. > >> > >> Has anybody checked whether we could do the cpu_down() on non-boot > >> CPU's in parallel? Right now we serialize the thing completely, with > > > > I thought Srivatsa S. Bhat had a patchset that did exactly that. > > Srivatsa? > > > > Thanks for the CC, Paul! Adding some more people to CC. > > Actually, my patchset was about removing stop_machine() from the CPU > offline path. > http://lwn.net/Articles/538819/ > > And here is the performance improvement I had measured in the version > prior to that: > http://article.gmane.org/gmane.linux.kernel/1435249 > > I'm planning to revive this patchset after the 3.10 merge window closes, > because it depends on doing a tree-wide sweep, and I think its a little > late to do it in time for the upcoming 3.10 merge window itself. > > Anyway, that's about removing stop_machine from CPU hotplug. > > Coming to bulk CPU hotplug, yes, I had ideas similar to what Russ suggested. > But I believe we can do more than that. > > As Russ pointed out, the notifiers are not thread-safe, so calling them > in parallel with different CPUs as arguments isn't going to work. > > So, first, we can convert all the CPU hotplug notifiers to take a cpumask > instead of a single CPU. So assuming that there are 'n' notifiers in total, > the number of function calls would become n, instead of n*1024. > But that itself most likely won't give us much benefit over the for-loop > that Russ has done in his patch, because it'll simply do longer processing > in each of those 'n' notifiers, by iterating over the cpumask inside each > notifier. > > Now comes the interesting thing: > > Consider a notifier chain that looks like this: > Priority 0: A->B->C->D > > We can't invoke say notifier callback A simultaneously on 2 CPUs with 2 > different hotcpus as argument. *However*, since A, B, C, D all (more or less) > belong to different subsystems, we can call A, B, C and D in parallel on > different CPUs. They won't even serialize amongst themselves because they > take locks (if any) of different subsystems. And since they are of same > priority, the ordering (A after B or B after A) doesn't matter as well. > > So with this, if we combine the idea I wrote above about giving a cpumask > to each of these notifiers to work with, we end up in this: > > CPU 0 CPU 1 CPU2 .... > A(cpumask) B(cpumask) C(cpumask) .... > > So, for example, the CPU_DOWN_PREPARE notification can be processed in parallel > on multiple CPUs at a time, for a given cpumask! That should definitely > give us a good speed-up. > > One more thing we have to note is that, there are 4 notifiers for taking a > CPU offline: > > CPU_DOWN_PREPARE > CPU_DYING > CPU_DEAD > CPU_POST_DEAD > > The first can be run in parallel as mentioned above. The second is run in > parallel in the stop_machine() phase as shown in Russ' patch. But the third > and fourth set of notifications all end up running only on CPU0, which will > again slow down things. > > So I suggest taking down the 1024 CPUs in multiple phases, like a binary search. > First, take 512 CPUs down, then 256 CPUs, then 128 CPUs etc. So at every bulk > CPU hotplug, we have enough online CPUs to handle the notifier load, and that > helps speed things up. Moreover, a handful of calls to stop_machine() is OK > because, stop_machine() takes progressively lesser and lesser time because > lesser CPUs are online on each iteration (and hence it reduces the > synchronization overhead of the stop-machine phase). > > The only downside to this whole idea of running the notifiers of a given > priority in parallel, is error handling - if a notifier fails, it would be > troublesome to rollback I guess. But if we forget that for a moment, we can > give this idea a try! > > Regards, > Srivatsa S. Bhat > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/
| |