Messages in this thread | | | Date | Wed, 2 Aug 2023 09:25:06 -0500 | Subject | Re: [Question] int3 instruction generates a #UD in SEV VM | From | Tom Lendacky <> |
| |
On 8/2/23 09:01, Sean Christopherson wrote: > On Wed, Aug 02, 2023, Wu Zongyo wrote: >> On Mon, Jul 31, 2023 at 11:45:29PM +0800, wuzongyong wrote: >>> >>> On 2023/7/31 23:03, Tom Lendacky wrote: >>>> On 7/31/23 09:30, Sean Christopherson wrote: >>>>> On Sat, Jul 29, 2023, wuzongyong wrote: >>>>>> Hi, >>>>>> I am writing a firmware in Rust to support SEV based on project td-shim[1]. >>>>>> But when I create a SEV VM (just SEV, no SEV-ES and no SEV-SNP) with the firmware, >>>>>> the linux kernel crashed because the int3 instruction in int3_selftest() cause a >>>>>> #UD. >>>>> >>>>> ... >>>>> >>>>>> BTW, if a create a normal VM without SEV by qemu & OVMF, the int3 instruction always generates a >>>>>> #BP. >>>>>> So I am confused now about the behaviour of int3 instruction, could anyone help to explain the behaviour? >>>>>> Any suggestion is appreciated! >>>>> >>>>> Have you tried my suggestions from the other thread[*]? >>> Firstly, I'm sorry for sending muliple mails with the same content. I thought the mails I sent previously >>> didn't be sent successfully. >>> And let's talk the problem here. >>>>> >>>>> : > > I'm curious how this happend. I cannot find any condition that would >>>>> : > > cause the int3 instruction generate a #UD according to the AMD's spec. >>>>> : >>>>> : One possibility is that the value from memory that gets executed diverges from the >>>>> : value that is read out be the #UD handler, e.g. due to patching (doesn't seem to >>>>> : be the case in this test), stale cache/tlb entries, etc. >>>>> : >>>>> : > > BTW, it worked nomarlly with qemu and ovmf. >>>>> : > >>>>> : > Does this happen every time you boot the guest with your firmware? What >>>>> : > processor are you running on? >>>>> : >>> Yes, every time. >>> The processor I used is EPYC 7T83. >>>>> : And have you ruled out KVM as the culprit? I.e. verified that KVM is NOT injecting >>>>> : a #UD. That obviously shouldn't happen, but it should be easy to check via KVM >>>>> : tracepoints. >>>> >>>> I have a feeling that KVM is injecting the #UD, but it will take instrumenting KVM to see which path the #UD is being injected from. >>>> >>>> Wu Zongyo, can you add some instrumentation to figure that out if the trace points towards KVM injecting the #UD? >>> Ok, I will try to do that. >> You're right. The #UD is injected by KVM. >> >> The path I found is: >> svm_vcpu_run >> svm_complete_interrupts >> kvm_requeue_exception // vector = 3 >> kvm_make_request >> >> vcpu_enter_guest >> kvm_check_and_inject_events >> svm_inject_exception >> svm_update_soft_interrupt_rip >> __svm_skip_emulated_instruction >> x86_emulate_instruction >> svm_can_emulate_instruction >> kvm_queue_exception(vcpu, UD_VECTOR) >> >> Does this mean a #PF intercept occur when the guest try to deliver a >> #BP through the IDT? But why? > > I doubt it's a #PF. A #NPF is much more likely, though it could be something > else entirely, but I'm pretty sure that would require bugs in both the host and > guest. > > What is the last exit recorded by trace_kvm_exit() before the #UD is injected?
I'm guessing it was a #NPF, too. Could it be related to the changes that went in around svm_update_soft_interrupt_rip()?
6ef88d6e36c2 ("KVM: SVM: Re-inject INT3/INTO instead of retrying the instruction")
Before this the !nrips check would prevent the call into svm_skip_emulated_instruction(). But now, there is a call to:
svm_update_soft_interrupt_rip() __svm_skip_emulated_instruction() kvm_emulate_instruction() x86_emulate_instruction() (passed a NULL insn pointer) kvm_can_emulate_insn() (passed a NULL insn pointer) svm_can_emulate_instruction() (passed NULL insn pointer)
Because it is an SEV guest, it ends up in the "if (unlikely(!insn))" path and injects the #UD.
Thanks, Tom
| |