Messages in this thread | | | Date | Wed, 9 Aug 2023 18:33:34 -0700 | From | Nathan Chancellor <> | Subject | Hang when booting guest kernels compiled with clang after SRSO mitigations |
| |
Hi Boris,
I updated my AMD 3990X workstation to a version of mainline that contains the SRSO mitigations and I am now seeing a hang when booting guest kernels built with clang in QEMU/KVM with an '-smp' value greater than one (I am just testing 'ARCH=x86_64 defconfig', nothing fancy). The host's kernel is built with GCC 13.2.0, in case that is relevant. The issue happens with all versions of clang that the kernel supports (11.x+). I do not see the issue with guest kernels built with GCC nor do I see the issue with '-smp 1', so it could be something that clang has done to the guest kernel that causes this but I figured I would report it early anyways.
With '-smp 4' (for example), I see
[ 0.102817] smpboot: CPU0: AMD Ryzen Threadripper 3990X 64-Core Processor (family: 0x17, model: 0x31, stepping: 0x0) ... [ 0.109778] smp: Bringing up secondary CPUs ... [ 0.110559] smpboot: x86: Booting SMP configuration:
then nothing after that, until timeout kills QEMU.
With '-smp 2', I can get all the way to userspace but it hangs when shutting down and I see what appears to be basically the same stack trace three times (I just included the last one):
Sent SIGKILL to all processes Requesting system poweroff [ 2.499704] ACPI: PM: Preparing to enter system sleep state S5 [ 2.500470] reboot: Power down ... [ 152.698101] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 152.698813] rcu: 0-...0: (9 ticks this GP) idle=556c/1/0x4000000000000000 softirq=333/335 fqs=29392 [ 152.699718] rcu: (detected by 1, t=147019 jiffies, g=-1003, q=2 ncpus=2) [ 152.700368] Sending NMI from CPU 1 to CPUs 0: [ 152.700795] NMI backtrace for cpu 0 [ 152.700799] CPU: 0 PID: 117 Comm: init Not tainted 6.5.0-rc5 #1 [ 152.700800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org 04/01/2014 [ 152.700802] RIP: 0010:default_send_IPI_allbutself+0x2a/0x60 [ 152.700806] Code: 83 ff 02 74 35 66 66 2e 0f 1f 84 00 00 00 00 00 f7 04 25 00 c3 5f ff 00 10 00 00 74 0f f3 90 f7 04 25 00 c3 5f ff 00 10 00 00 <75> f1 81 cf 00 00 0c 00 89 3c 25 00 c3 5f ff c3 48 8b 05 d7 2c 61 [ 152.700807] RSP: 0018:ffff99e58027fdb0 EFLAGS: 00000082 [ 152.700808] RAX: ffffffff8426ab50 RBX: 0000000000000000 RCX: 0000000000000001 [ 152.700809] RDX: ffffffffff5fb000 RSI: 00000000000000f8 RDI: 00000000000000f8 [ 152.700810] RBP: 0000000000000001 R08: 0000000000000000 R09: ffffffff84482120 [ 152.700810] R10: 0000000000000000 R11: ffffffff82c57f50 R12: 000000004321fedc [ 152.700811] R13: fffffffffee1dead R14: 0000000000000000 R15: 0000000028121969 [ 152.700813] FS: 00007f1eb36696a0(0000) GS:ffff978b9f000000(0000) knlGS:0000000000000000 [ 152.700814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 152.700815] CR2: 00007f09d4a8a360 CR3: 00000000024c4000 CR4: 0000000000350ef0 [ 152.700817] Call Trace: [ 152.700819] <NMI> [ 152.700820] ? nmi_cpu_backtrace+0xde/0x110 [ 152.700822] ? nmi_cpu_backtrace_handler+0x8/0x10 [ 152.700823] ? nmi_handle+0x69/0x150 [ 152.700824] ? default_do_nmi+0x43/0x1d0 [ 152.700826] ? exc_nmi+0xbc/0x130 [ 152.700827] ? end_repeat_nmi+0x16/0x67 [ 152.700836] ? default_send_IPI_single+0x30/0x30 [ 152.700838] ? default_send_IPI_allbutself+0x2a/0x60 [ 152.700839] ? default_send_IPI_allbutself+0x2a/0x60 [ 152.700840] ? default_send_IPI_allbutself+0x2a/0x60 [ 152.700840] </NMI> [ 152.700841] <TASK> [ 152.700841] native_stop_other_cpus+0x7d/0x1f0 [ 152.700843] native_machine_shutdown+0x17/0x40 [ 152.700844] native_machine_power_off+0x24/0x30 [ 152.700846] __se_sys_reboot+0x221/0x230 [ 152.700848] do_syscall_64+0x31/0x50 [ 152.700849] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 152.700851] RIP: 0033:0x7f1eb35ceffb [ 152.700853] Code: ff 76 10 48 8b 15 95 ee 07 00 f7 d8 64 89 02 48 83 c8 ff c3 48 63 d7 be 69 19 12 28 b8 a9 00 00 00 48 c7 c7 ad de e1 fe 0f 05 <48> 3d 00 f0 ff ff 76 10 48 8b 15 66 ee 07 00 f7 d8 64 89 02 48 83 [ 152.700854] RSP: 002b:00007fff81094238 EFLAGS: 00000246 ORIG_RAX: 00000000000000a9 [ 152.700855] RAX: ffffffffffffffda RBX: 000000000000000c RCX: 00007f1eb35ceffb [ 152.700855] RDX: 000000004321fedc RSI: 0000000028121969 RDI: fffffffffee1dead [ 152.700856] RBP: 000000004321fedc R08: 8080808080808080 R09: 6b6470ff65656e71 [ 152.700856] R10: 0000000000000008 R11: 0000000000000246 R12: 00007fff81095fc2 [ 152.700857] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 152.700858] </TASK>
I tested the other mitigation options for SRSO and it seems the only one that has problems is safe-ret (I don't think there is a microcode update for this machine yet, so I did not both trying microcode).
spec_rstack_overflow=off: No issue spec_rstack_overflow=safe-ret: Has issue spec_rstack_overflow=ibpb: No issue spec_rstack_overflow=ibpb-vmexit: No issue
My QEMU command, in case it is relevant, the rootfs is available at https://github.com/ClangBuiltLinux/boot-utils/releases/tag/20230707-182910.
$ qemu-system-x86_64 \ -display none \ -nodefaults \ -d unimp,guest_errors \ -append 'console=ttyS0 earlycon=uart8250,io,0x3f8' \ -kernel arch/x86/boot/bzImage \ -initrd rootfs.cpio \ -cpu host \ -enable-kvm \ -m 512m \ -smp 4 \ -serial mon:stdio
If there is any other information I can provide or patches I can test, I am more than happy to do so. Should you need access to a clang toolchain for building the guest kernel (if you did not want to bother installing one from your distro), I have ones available on kernel.org similar to the GCC ones that should work fine:
https://mirrors.edge.kernel.org/pub/tools/llvm/
Cheers, Nathan
| |