Messages in this thread | | | From | Rajender M <> | Subject | Re: Performance impact in networking data path tests in Linux 5.5 Kernel | Date | Wed, 26 Feb 2020 11:45:34 +0000 |
| |
Thanks for your response, Vincent. Just curious to know, if there are any room for optimizing the additional CPU cost.
On 26/02/20, 3:18 PM, "Vincent Guittot" <vincent.guittot@linaro.org> wrote:
Hi Rajender,
On Tue, 25 Feb 2020 at 06:46, Rajender M <manir@vmware.com> wrote: > > As part of VMware's performance regression testing for Linux Kernel upstream > releases, when comparing Linux 5.5 kernel against Linux 5.4 kernel, we noticed > 20% improvement in networking throughput performance at the cost of a 30% > increase in the CPU utilization. Thanks for testing and sharing results with us. It's always interesting to get feedbacks from various tests cases > > After performing the bisect between 5.4 and 5.5, we identified the root cause > of this behaviour to be a scheduling change from Vincent Guittot's > 2ab4092fc82d ("sched/fair: Spread out tasks evenly when not overloaded"). > > The impacted testcases are TCP_STREAM SEND & RECV – on both small > (8K socket & 256B message) & large (64K socket & 16K message) packet sizes. > > We backed out Vincent's commit & reran our networking tests and found that > the performance were similar to 5.4 kernel - improvements in networking tests > were no more. > > In our current network performance testing, we use Intel 10G NIC to evaluate > all Linux Kernel releases. In order to confirm that the impact is also seen in > higher bandwidth NIC, we repeated the same test cases with Intel 40G and > we were able to reproduce the same behaviour - 25% improvements in > throughput with 10% more CPU consumption. > > The overall results indicate that the new scheduler change has introduced > much better network throughput performance at the cost of incremental > CPU usage. This can be seen as expected behavior because now the > TCP streams are evenly spread across all the CPUs and eventually drives > more network packets, with additional CPU consumption. > > > We have also confirmed this theory by parsing the ESX stats for 5.4 and 5.5 > kernels in a 4vCPU VM running 8 TCP streams - as shown below; > > 5.4 kernel: > "2132149": {"id": 2132149, "used": 94.37, "ready": 0.01, "cstp": 0.00, "name": "vmx-vcpu-0:rhel7x64-0", > "2132151": {"id": 2132151, "used": 0.13, "ready": 0.00, "cstp": 0.00, "name": "vmx-vcpu-1:rhel7x64-0", > "2132152": {"id": 2132152, "used": 9.07, "ready": 0.03, "cstp": 0.00, "name": "vmx-vcpu-2:rhel7x64-0", > "2132153": {"id": 2132153, "used": 34.77, "ready": 0.01, "cstp": 0.00, "name": "vmx-vcpu-3:rhel7x64-0", > > 5.5 kernel: > "2132041": {"id": 2132041, "used": 55.70, "ready": 0.01, "cstp": 0.00, "name": "vmx-vcpu-0:rhel7x64-0", > "2132043": {"id": 2132043, "used": 47.53, "ready": 0.01, "cstp": 0.00, "name": "vmx-vcpu-1:rhel7x64-0", > "2132044": {"id": 2132044, "used": 77.81, "ready": 0.00, "cstp": 0.00, "name": "vmx-vcpu-2:rhel7x64-0", > "2132045": {"id": 2132045, "used": 57.11, "ready": 0.02, "cstp": 0.00, "name": "vmx-vcpu-3:rhel7x64-0", > > Note, "used %" in above stats for 5.5 kernel is evenly distributed across all vCPUs. > > On the whole, this change should be seen as a significant improvement for > most customers. > > Rajender M > Performance Engineering > VMware, Inc. >
| |