lkml.org 
[lkml]   [2014]   [Sep]   [22]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [RFC][PATCH 0/6] fix topology for multi-NUMA-node CPUs
On 9/22/2014 9:33 AM, Aravind Gopalakrishnan wrote:
>
> This is a big fat RFC. It takes quite a few liberties with the
> multi-core topology level that I'm not completely comfortable
> with.
>
> It has only been tested lightly.
>
> Full dmesg for a Cluster-on-Die system with this set applied,
> and sched_debug on the command-line is here:
>
> http://sr71.net/~dave/intel/full-dmesg-hswep-20140917.txt
> <http://sr71.net/%7Edave/intel/full-dmesg-hswep-20140917.txt>
>
> ---
>
> I'm getting the spew below when booting with Haswell (Xeon
> E5-2699 v3) CPUs and the "Cluster-on-Die" (CoD) feature enabled
> in the BIOS. It seems similar to the issue that some folks from
> AMD ran in to on their systems and addressed in this commit:
>
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=161270fc1f9ddfc17154e0d49291472a9cdef7db
>
> Both these Intel and AMD systems break an assumption which is
> being enforced by topology_sane(): a socket may not contain more
> than one NUMA node.
>
> AMD special-cased their system by looking for a cpuid flag. The
> Intel mode is dependent on BIOS options and I do not know of a
> way which it is enumerated other than the tables being parsed
> during the CPU bringup process.
>
> This also fixes sysfs because CPUs with the same 'physical_package_id'
> in /sys/devices/system/cpu/cpu*/topology/ are not listed together
> in the same 'core_siblings_list'. This violates a statement from
> Documentation/ABI/testing/sysfs-devices-system-cpu:
>
> core_siblings: internal kernel map of cpu#'s hardware threads
> within the same physical_package_id.
>
> core_siblings_list: human-readable list of the logical CPU
> numbers within the same physical_package_id as cpu#.
>
> The sysfs effects here cause an issue with the hwloc tool where
> it gets confused and thinks there are more sockets than are
> physically present.
>
> Before this set, there are two packages:
>
> # cd /sys/devices/system/cpu/
> # cat cpu*/topology/physical_package_id | sort | uniq -c
> 18 0
> 18 1
>
> But 4 _sets_ of core siblings:
>
> # cat cpu*/topology/core_siblings_list | sort | uniq -c
> 9 0-8
> 9 18-26
> 9 27-35
> 9 9-17
>
> After this set, there are only 2 sets of core siblings, which
> is what we expect for a 2-socket system.
>
> # cat cpu*/topology/physical_package_id | sort | uniq -c
> 18 0
> 18 1
> # cat cpu*/topology/core_siblings_list | sort | uniq -c
> 18 0-17
> 18 18-35
>
>
> Example spew:
> ...
> NMI watchdog: enabled on all CPUs, permanently consumes one
> hw-PMU counter.
> #2 #3 #4 #5 #6 #7 #8
> .... node #1, CPUs: #9
> ------------[ cut here ]------------
> WARNING: CPU: 9 PID: 0 at
> /home/ak/hle/linux-hle-2.6/arch/x86/kernel/smpboot.c:306
> topology_sane.isra.2+0x74/0x90()
> sched: CPU #9's mc-sibling CPU #0 is not on the same node!
> [node: 1 != 0]. Ignoring dependency.
> Modules linked in:
> CPU: 9 PID: 0 Comm: swapper/9 Not tainted
> 3.17.0-rc1-00293-g8e01c4d-dirty #631
> Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> GRNDSDP1.86B.0036.R05.1407140519 07/14/2014
> 0000000000000009 ffff88046ddabe00 ffffffff8172e485
> ffff88046ddabe48
> ffff88046ddabe38 ffffffff8109691d 000000000000b001
> 0000000000000009
> ffff88086fc12580 000000000000b020 0000000000000009
> ffff88046ddabe98
> Call Trace:
> [<ffffffff8172e485>] dump_stack+0x45/0x56
> [<ffffffff8109691d>] warn_slowpath_common+0x7d/0xa0
> [<ffffffff8109698c>] warn_slowpath_fmt+0x4c/0x50
> [<ffffffff81074f94>] topology_sane.isra.2+0x74/0x90
> [<ffffffff8107530e>] set_cpu_sibling_map+0x31e/0x4f0
> [<ffffffff8107568d>] start_secondary+0x1ad/0x240
> ---[ end trace 3fe5f587a9fcde61 ]---
> #10 #11 #12 #13 #14 #15 #16 #17
> .... node #2, CPUs: #18 #19 #20 #21 #22 #23 #24 #25 #26
> .... node #3, CPUs: #27 #28 #29 #30 #31 #32 #33 #34 #35


Hi,
I looked at the topology info from sysfs both w/ and w/o the patch
series and they are identical.
So, the patches seem to work fine on an AMD MCM part.

Tested-by: Aravind Gopalakrishnan <Aravind.Gopalakrishnan@amd.com>

Thanks,
-Aravind.


\
 
 \ /
  Last update: 2014-09-22 18:41    [W:0.946 / U:0.448 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site