Messages in this thread |  | | Subject | Re: [PATCH -tip 00/32] Core scheduling (v9) | From | "Ning, Hongyu" <> | Date | Thu, 3 Dec 2020 14:16:28 +0800 |
| |
On 2020/11/24 23:08, Joel Fernandes wrote: >>> >>> Core-Scheduling >>> =============== >>> Enclosed is series v9 of core scheduling. >>> v9 is rebased on tip/master (fe4adf6f92c4 ("Merge branch 'irq/core'")).. >>> I hope that this version is acceptable to be merged (pending any new review >>> comments that arise) as the main issues in the past are all resolved: >>> 1. Vruntime comparison. >>> 2. Documentation updates. >>> 3. CGroup and per-task interface developed by Google and Oracle. >>> 4. Hotplug fixes. >>> Almost all patches also have Reviewed-by or Acked-by tag. See below for full >>> list of changes in v9. >>> >>> Introduction of feature >>> ======================= >>> Core scheduling is a feature that allows only trusted tasks to run >>> concurrently on cpus sharing compute resources (eg: hyperthreads on a >>> core). The goal is to mitigate the core-level side-channel attacks >>> without requiring to disable SMT (which has a significant impact on >>> performance in some situations). Core scheduling (as of v7) mitigates >>> user-space to user-space attacks and user to kernel attack when one of >>> the siblings enters the kernel via interrupts or system call. >>> >>> By default, the feature doesn't change any of the current scheduler >>> behavior. The user decides which tasks can run simultaneously on the >>> same core (for now by having them in the same tagged cgroup). When a tag >>> is enabled in a cgroup and a task from that cgroup is running on a >>> hardware thread, the scheduler ensures that only idle or trusted tasks >>> run on the other sibling(s). Besides security concerns, this feature can >>> also be beneficial for RT and performance applications where we want to >>> control how tasks make use of SMT dynamically. >>> >>> Both a CGroup and Per-task interface via prctl(2) are provided for configuring >>> core sharing. More details are provided in documentation patch. Kselftests are >>> provided to verify the correctness/rules of the interface. >>> >>> Testing >>> ======= >>> ChromeOS testing shows 300% improvement in keypress latency on a Google >>> docs key press with Google hangout test (the maximum latency drops from 150ms >>> to 50ms for keypresses). >>> >>> Julien: TPCC tests showed improvements with core-scheduling as below. With kernel >>> protection enabled, it does not show any regression. Possibly ASI will improve >>> the performance for those who choose kernel protection (can be toggled through >>> sched_core_protect_kernel sysctl). >>> average stdev diff >>> baseline (SMT on) 1197.272 44.78312824 >>> core sched ( kernel protect) 412.9895 45.42734343 -65.51% >>> core sched (no kernel protect) 686.6515 71.77756931 -42.65% >>> nosmt 408.667 39.39042872 -65.87% >>> (Note these results are from v8). >>> >>> Vineeth tested sysbench and does not see any regressions. >>> Hong and Aubrey tested v9 and see results similar to v8. There is a known issue >>> with uperf that does regress. This appears to be because of ksoftirq heavily >>> contending with other tasks on the core. The consensus is this can be improved >>> in the future. >>> >>> Changes in v9 >>> ============= >>> - Note that the vruntime snapshot change is written in 2 patches to show the >>> progression of the idea and prevent merge conflicts: >>> sched/fair: Snapshot the min_vruntime of CPUs on force idle >>> sched: Improve snapshotting of min_vruntime for CGroups >>> Same with the RT priority inversion change: >>> sched: Fix priority inversion of cookied task with sibling >>> sched: Improve snapshotting of min_vruntime for CGroups >>> - Disable coresched on certain AMD HW. >>>
Adding workloads and negative case test results for core scheduling v9 posted:
- kernel under test: -- coresched community v9 posted from https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/log/?h=sched/coresched-v9-posted (tag: sched/coresched-v9-posted) -- latest commit: d48636e429de (HEAD -> coresched-v9-posted, tag: sched/coresched-v9-posted) sched: Debug bits... -- coresched=on kernel parameter applied - workloads: -- A. sysbench cpu (192 threads) + sysbench cpu (192 threads) -- B. sysbench cpu (192 threads) + sysbench mysql (192 threads, mysqld forced into the same cgroup) -- C. uperf netperf.xml (192 threads over TCP or UDP protocol separately) -- D. will-it-scale context_switch via pipe (192 threads) - negative case: -- A. continuously toggle cpu.core_tag, during full loading uperf workload running with cs_on -- B. continuously toggle smt setting via /sys/devices/system/cpu/smt/control, during full loading uperf workload running with cs_on -- C. continuously cgroup switch between cs_on cgroup and cs_off cgroup via cgclassify, during full loading uperf workload running - test machine setup: CPU(s): 192 On-line CPU(s) list: 0-191 Thread(s) per core: 2 Core(s) per socket: 48 Socket(s): 2 NUMA node(s): 4 - test results of workloads, no obvious performance drop compared to community v8 build: -- workload A: +----------------------+------+----------------------+------------------------+ | workloads | ** | sysbench cpu * 192 | sysbench cpu * 192 | +======================+======+======================+========================+ | cgroup | ** | cg_sysbench_cpu_0 | cg_sysbench_cpu_1 | +----------------------+------+----------------------+------------------------+ | record_item | ** | Tput_avg (events/s) | Tput_avg (events/s) | +----------------------+------+----------------------+------------------------+ | coresched_normalized | ** | 0.97 | 1.02 | +----------------------+------+----------------------+------------------------+ | default_normalized | ** | 1.00 | 1.00 | +----------------------+------+----------------------+------------------------+ | smtoff_normalized | ** | 0.60 | 0.60 | +----------------------+------+----------------------+------------------------+
-- workload B: +----------------------+------+----------------------+------------------------+ | workloads | ** | sysbench cpu * 192 | sysbench mysql * 192 | +======================+======+======================+========================+ | cgroup | ** | cg_sysbench_cpu_0 | cg_sysbench_mysql_0 | +----------------------+------+----------------------+------------------------+ | record_item | ** | Tput_avg (events/s) | Tput_avg (events/s) | +----------------------+------+----------------------+------------------------+ | coresched_normalized | ** | 0.94 | 0.88 | +----------------------+------+----------------------+------------------------+ | default_normalized | ** | 1.00 | 1.00 | +----------------------+------+----------------------+------------------------+ | smtoff_normalized | ** | 0.56 | 0.84 | +----------------------+------+----------------------+------------------------+
-- workload C: +----------------------+------+---------------------------+---------------------------+ | workloads | ** | uperf netperf TCP * 192 | uperf netperf UDP * 192 | +======================+======+===========================+===========================+ | cgroup | ** | cg_uperf | cg_uperf | +----------------------+------+---------------------------+---------------------------+ | record_item | ** | Tput_avg (Gb/s) | Tput_avg (Gb/s) | +----------------------+------+---------------------------+---------------------------+ | coresched_normalized | ** | 0.64 | 0.68 | +----------------------+------+---------------------------+---------------------------+ | default_normalized | ** | 1.00 | 1.00 | +----------------------+------+---------------------------+---------------------------+ | smtoff_normalized | ** | 0.92 | 0.89 | +----------------------+------+---------------------------+---------------------------+
-- workload D: +----------------------+------+-------------------------------+ | workloads | ** | will-it-scale * 192 | | | | (pipe based context_switch) | +======================+======+===============================+ | cgroup | ** | cg_will-it-scale | +----------------------+------+-------------------------------+ | record_item | ** | threads_avg | +----------------------+------+-------------------------------+ | coresched_normalized | ** | 0.30 | +----------------------+------+-------------------------------+ | default_normalized | ** | 1.00 | +----------------------+------+-------------------------------+ | smtoff_normalized | ** | 0.87 | +----------------------+------+-------------------------------+
-- notes on record_item: * coresched_normalized: smton, cs enabled, test result normalized by default value * default_normalized: smton, cs disabled, test result normalized by default value * smtoff_normalized: smtoff, test result normalized by default value
- test results of negative case, all as expected, no kernel panic or system hang observed
Hongyu
|  |