Messages in this thread | | | Subject | Re: [PATCH v3 0/9] Parallel CPU bringup for x86_64 | From | Tom Lendacky <> | Date | Fri, 17 Dec 2021 11:48:49 -0600 |
| |
On 12/16/21 6:13 PM, David Woodhouse wrote: > On Thu, 2021-12-16 at 16:52 -0600, Tom Lendacky wrote: >> On baremetal, I haven't seen an issue. This only seems to have a problem >> with Qemu/KVM. >> >> With 191f08997577 I could boot without issues with and without the >> no_parallel_bringup. Only after I applied e78fa57dd642 did the failure happen. >> >> With e78fa57dd642 I could boot 64 vCPUs pretty consistently, but when I >> jumped to 128 vCPUs it failed again. When I moved the series to >> df9726cb7178, then 64 vCPUs also failed pretty consistently. >> >> Strange thing is it is random. Sometimes (rarely) it works on the first >> boot and then sometimes it doesn't, at which point it will reset and >> reboot 3 or 4 times and then make it past the failure and fully boot. > > Hm, some of that is just artifacts of timing, I'm sure. But now I'm > staring at the way that early_setup_idt() can run in parallel on all > CPUs, rewriting bringup_idt_descr and loading it. > > To start with, let's try unlocking the trampoline_lock much later, > after cpu_init_exception_handling() has loaded the real IDT. > > I think we can probably make secondaries load the real IDT early and > never use bringup_idt_descr at all, can't we? But let's see if this > makes it go away, to start with... >
This still fails. I ran with -d cpu_reset on the command line and will forward the full log to you. I ran "grep "[ER]IP=" stderr.log | uniq -c" and got:
128 EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=0 SMM=0 HLT=0 128 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 These are before running any of the vCPUs.
1 RIP=ffffffff810705c6 RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 This is where vCPU0 is at the time of the reset. This address tends to be different all the time and so I think it is just where it happens to be when the reset occurs and isn't contributing to the reset. 5 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 1 RIP=ffffffff8104af06 RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 15 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 These are some of the APs and all are in wait_for_master_cpu().
1 EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0 This seems ok because: CS =9900 00099000 0000ffff 00009b00 So likely in the trampoline code.
1 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 This one seems odd... could it be the one causing the reset? CS =f000 ffff0000 0000ffff 00009a00
3 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 2 EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0 99 EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
Thanks, Tom
| |