Messages in this thread | | | Date | Mon, 22 Sep 2014 10:54:24 -0500 | From | Aravind Gopalakrishnan <> | Subject | Re: [RFC][PATCH 0/6] fix topology for multi-NUMA-node CPUs |
| |
On 9/22/2014 9:33 AM, Aravind Gopalakrishnan wrote: > > This is a big fat RFC. It takes quite a few liberties with the > multi-core topology level that I'm not completely comfortable > with. > > It has only been tested lightly. > > Full dmesg for a Cluster-on-Die system with this set applied, > and sched_debug on the command-line is here: > > http://sr71.net/~dave/intel/full-dmesg-hswep-20140917.txt > <http://sr71.net/%7Edave/intel/full-dmesg-hswep-20140917.txt> > > --- > > I'm getting the spew below when booting with Haswell (Xeon > E5-2699 v3) CPUs and the "Cluster-on-Die" (CoD) feature enabled > in the BIOS. It seems similar to the issue that some folks from > AMD ran in to on their systems and addressed in this commit: > > http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=161270fc1f9ddfc17154e0d49291472a9cdef7db > > Both these Intel and AMD systems break an assumption which is > being enforced by topology_sane(): a socket may not contain more > than one NUMA node. > > AMD special-cased their system by looking for a cpuid flag. The > Intel mode is dependent on BIOS options and I do not know of a > way which it is enumerated other than the tables being parsed > during the CPU bringup process. > > This also fixes sysfs because CPUs with the same 'physical_package_id' > in /sys/devices/system/cpu/cpu*/topology/ are not listed together > in the same 'core_siblings_list'. This violates a statement from > Documentation/ABI/testing/sysfs-devices-system-cpu: > > core_siblings: internal kernel map of cpu#'s hardware threads > within the same physical_package_id. > > core_siblings_list: human-readable list of the logical CPU > numbers within the same physical_package_id as cpu#. > > The sysfs effects here cause an issue with the hwloc tool where > it gets confused and thinks there are more sockets than are > physically present. > > Before this set, there are two packages: > > # cd /sys/devices/system/cpu/ > # cat cpu*/topology/physical_package_id | sort | uniq -c > 18 0 > 18 1 > > But 4 _sets_ of core siblings: > > # cat cpu*/topology/core_siblings_list | sort | uniq -c > 9 0-8 > 9 18-26 > 9 27-35 > 9 9-17 > > After this set, there are only 2 sets of core siblings, which > is what we expect for a 2-socket system. > > # cat cpu*/topology/physical_package_id | sort | uniq -c > 18 0 > 18 1 > # cat cpu*/topology/core_siblings_list | sort | uniq -c > 18 0-17 > 18 18-35 > > > Example spew: > ... > NMI watchdog: enabled on all CPUs, permanently consumes one > hw-PMU counter. > #2 #3 #4 #5 #6 #7 #8 > .... node #1, CPUs: #9 > ------------[ cut here ]------------ > WARNING: CPU: 9 PID: 0 at > /home/ak/hle/linux-hle-2.6/arch/x86/kernel/smpboot.c:306 > topology_sane.isra.2+0x74/0x90() > sched: CPU #9's mc-sibling CPU #0 is not on the same node! > [node: 1 != 0]. Ignoring dependency. > Modules linked in: > CPU: 9 PID: 0 Comm: swapper/9 Not tainted > 3.17.0-rc1-00293-g8e01c4d-dirty #631 > Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS > GRNDSDP1.86B.0036.R05.1407140519 07/14/2014 > 0000000000000009 ffff88046ddabe00 ffffffff8172e485 > ffff88046ddabe48 > ffff88046ddabe38 ffffffff8109691d 000000000000b001 > 0000000000000009 > ffff88086fc12580 000000000000b020 0000000000000009 > ffff88046ddabe98 > Call Trace: > [<ffffffff8172e485>] dump_stack+0x45/0x56 > [<ffffffff8109691d>] warn_slowpath_common+0x7d/0xa0 > [<ffffffff8109698c>] warn_slowpath_fmt+0x4c/0x50 > [<ffffffff81074f94>] topology_sane.isra.2+0x74/0x90 > [<ffffffff8107530e>] set_cpu_sibling_map+0x31e/0x4f0 > [<ffffffff8107568d>] start_secondary+0x1ad/0x240 > ---[ end trace 3fe5f587a9fcde61 ]--- > #10 #11 #12 #13 #14 #15 #16 #17 > .... node #2, CPUs: #18 #19 #20 #21 #22 #23 #24 #25 #26 > .... node #3, CPUs: #27 #28 #29 #30 #31 #32 #33 #34 #35
Hi, I looked at the topology info from sysfs both w/ and w/o the patch series and they are identical. So, the patches seem to work fine on an AMD MCM part.
Tested-by: Aravind Gopalakrishnan <Aravind.Gopalakrishnan@amd.com>
Thanks, -Aravind.
| |