Messages in this thread Patch in this message | | | Subject | [PATCH] smp/hotplug, x86/vmware: Put offline vCPUs in halt instead of mwait | From | "Srivatsa S. Bhat" <> | Date | Thu, 21 Jul 2022 13:44:33 -0700 |
| |
From: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu>
VMware ESXi allows enabling a passthru mwait CPU-idle state in the guest using the following VMX option:
monitor_control.mwait_in_guest = "TRUE"
This lets a vCPU in mwait to remain in guest context (instead of yielding to the hypervisor via a VMEXIT), which helps speed up wakeups from idle.
However, this runs into problems with CPU hotplug, because the Linux CPU offline path prefers to put the vCPU-to-be-offlined in mwait state, whenever mwait is available. As a result, since a vCPU in mwait remains in guest context and does not yield to the hypervisor, an offline vCPU *appears* to be 100% busy as viewed from ESXi, which prevents the hypervisor from running other vCPUs or workloads on the corresponding pCPU (particularly when vCPU - pCPU mappings are statically defined by the user). [ Note that such a vCPU is not actually busy spinning though; it remains in mwait idle state in the guest ].
Fix this by overriding the CPU offline play_dead() callback for VMware hypervisor, by putting the CPU in halt state (which actually yields to the hypervisor), even if mwait support is available.
Signed-off-by: Srivatsa S. Bhat (VMware) <srivatsa@csail.mit.edu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Borislav Petkov <bp@alien8.de> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Alexey Makhalov <amakhalov@vmware.com> Cc: Juergen Gross <jgross@suse.com> Cc: x86@kernel.org Cc: VMware PV-Drivers Reviewers <pv-drivers@vmware.com> ---
arch/x86/kernel/cpu/vmware.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c index c04b933f48d3..420e359ed9bb 100644 --- a/arch/x86/kernel/cpu/vmware.c +++ b/arch/x86/kernel/cpu/vmware.c @@ -27,6 +27,7 @@ #include <linux/clocksource.h> #include <linux/cpu.h> #include <linux/reboot.h> +#include <linux/tboot.h> #include <linux/static_call.h> #include <asm/div64.h> #include <asm/x86_init.h> @@ -312,6 +313,21 @@ static int vmware_cpu_down_prepare(unsigned int cpu) local_irq_enable(); return 0; } + +static void vmware_play_dead(void) +{ + play_dead_common(); + tboot_shutdown(TB_SHUTDOWN_WFS); + + /* + * Put the vCPU going offline in halt instead of mwait (even + * if mwait support is available), to make sure that the + * offline vCPU yields to the hypervisor (which may not happen + * with mwait, for example, if the guest's VMX is configured + * to retain the vCPU in guest context upon mwait). + */ + hlt_play_dead(); +} #endif static __init int activate_jump_labels(void) @@ -349,6 +365,7 @@ static void __init vmware_paravirt_ops_setup(void) #ifdef CONFIG_SMP smp_ops.smp_prepare_boot_cpu = vmware_smp_prepare_boot_cpu; + smp_ops.play_dead = vmware_play_dead; if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "x86/vmware:online", vmware_cpu_online,
| |