Messages in this thread | | | From | sat <> | Subject | low performance of kernel build with quite a few simultaneous jobs | Date | Fri, 24 Aug 2018 00:50:43 +0000 |
| |
Hi,
I found that kernel build speeds are very low with quite a few `n` for `make -j<n>`, from one to the number of logical CPUs(LCPUs) on my EPYC server.
The speeds of `make -j<n>` look fine with `n` roughly from 1 to 24, and from 64 to 96, but are slow with other `n`s (please refer to the attached images). I suspect that there is a task scheduling problem about CPUs with complicated topology like EPYC . Plus, it can't be explained only by the node distance problem because this behavior doesn't disappear with enabling memory interleave.
Could you guess any reason of this strange behavior? I ask you because you wrote the patch for better scheduling of recent AMD CPUs, commit 051f3ca02e46432c0965e8948f00c07d8a2f09c0 ("sched/topology: Introduce NUMA identity node sched domain, ").
Here is the detailed information.
** Hardware Engironment
- server: Super Micro AS-1023US-TR4 - CPU: EPYC 7451 x 2 (48 cores、96 threads) - RAM: MEM-DR432L-HL01-ER26 (32GB * 16, totally 512GB)
** Software Environment
- OS: Ubuntu 18.04 - kernel: v4.18.0 - build target kernel: v4.18.0
** Test load
Run the following commands.
``` make defconfig for ((i=1:1<`# of online LCPUs``;i++)) ; do make clean && ( time make -j$i ) >result.txt done ```
I measured the above mentioned data for the following settings.
- all: 96 LCPUS: online whole LCPUs - 1sock: 48 LCPUs: online only the LCPUS in one sockets - 1die: 12 LCPUs: ... in one die - 1ccx: 6 LCPUs: ... in one ccx
** result
Please refer to the attached files.
- 4.18.0-no-interleave.png: Setting no interleave on UEFI
node distances got by `numactl -H` is as follows.
``` node 0 1 2 3 4 5 6 7 0: 10 16 16 16 32 32 32 32 1: 16 10 16 16 32 32 32 32 2: 16 16 10 16 32 32 32 32 3: 16 16 16 10 32 32 32 32 4: 32 32 32 32 10 16 16 16 5: 32 32 32 32 16 10 16 16 6: 32 32 32 32 16 16 10 16 7: 32 32 32 32 16 16 16 10 ```
- 4.18.0-hw-interleave.png: Setting interleave on UEFI.
node distance is all 10 in this case.
Here x axis is `n` for `make -j<n>` and y axis is "real" value of time command for above mentioned kernel build over `n`.
In the attached images, all cases besides 1ccx looks strange.
Thanks, Satoru Takeuchi[unhandled content-type:image/png][unhandled content-type:image/png] | |