lkml.org 
[lkml]   [2016]   [Dec]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    From
    Subject[PATCH v3 4/4] x86/asm: Rewrite sync_core() to use IRET-to-self
    Date
    Aside from being excessively slow, CPUID is problematic: Linux runs
    on a handful of CPUs that don't have CPUID. Use IRET-to-self
    instead. IRET-to-self works everywhere, so it makes testing easy.

    For reference, On my laptop, IRET-to-self is ~110ns,
    CPUID(eax=1, ecx=0) is ~83ns on native and very very slow under KVM,
    and MOV-to-CR2 is ~42ns.

    While we're at it: sync_core() serves a very specific purpose.
    Document it.

    Cc: "H. Peter Anvin" <hpa@zytor.com>
    Signed-off-by: Andy Lutomirski <luto@kernel.org>
    ---
    arch/x86/include/asm/processor.h | 77 ++++++++++++++++++++++++++++------------
    1 file changed, 55 insertions(+), 22 deletions(-)

    diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
    index 64fbc937d586..201a956e345f 100644
    --- a/arch/x86/include/asm/processor.h
    +++ b/arch/x86/include/asm/processor.h
    @@ -590,33 +590,66 @@ static __always_inline void cpu_relax(void)

    #define cpu_relax_lowlatency() cpu_relax()

    -/* Stop speculative execution and prefetching of modified code. */
    +/*
    + * This function forces the icache and prefetched instruction stream to
    + * catch up with reality in two very specific cases:
    + *
    + * a) Text was modified using one virtual address and is about to be executed
    + * from the same physical page at a different virtual address.
    + *
    + * b) Text was modified on a different CPU, may subsequently be
    + * executed on this CPU, and you want to make sure the new version
    + * gets executed. This generally means you're calling this in a IPI.
    + *
    + * If you're calling this for a different reason, you're probably doing
    + * it wrong.
    + */
    static inline void sync_core(void)
    {
    - int tmp;
    -
    -#ifdef CONFIG_X86_32
    /*
    - * Do a CPUID if available, otherwise do a jump. The jump
    - * can conveniently enough be the jump around CPUID.
    + * There are quite a few ways to do this. IRET-to-self is nice
    + * because it works on every CPU, at any CPL (so it's compatible
    + * with paravirtualization), and it never exits to a hypervisor.
    + * The only down sides are that it's a bit slow (it seems to be
    + * a bit more than 2x slower than the fastest options) and that
    + * it unmasks NMIs. The "push %cs" is needed because, in
    + * paravirtual environments, __KERNEL_CS may not be a valid CS
    + * value when we do IRET directly.
    + *
    + * In case NMI unmasking or performance every becomes a problem,
    + * the next best option appears to be MOV-to-CR2 and an
    + * unconditional jump. That sequence also works on all CPUs,
    + * but it will fault at CPL3.
    + *
    + * CPUID is the conventional way, but it's nasty: it doesn't
    + * exist on some 486-like CPUs, and it usually exits to a
    + * hypervisor.
    */
    - asm volatile("cmpl %2,%1\n\t"
    - "jl 1f\n\t"
    - "cpuid\n"
    - "1:"
    - : "=a" (tmp)
    - : "rm" (boot_cpu_data.cpuid_level), "ri" (0), "0" (1)
    - : "ebx", "ecx", "edx", "memory");
    + register void *__sp asm(_ASM_SP);
    +
    +#ifdef CONFIG_X86_32
    + asm volatile (
    + "pushfl\n\t"
    + "pushl %%cs\n\t"
    + "pushl $1f\n\t"
    + "iret\n\t"
    + "1:"
    + : "+r" (__sp) : : "cc", "memory");
    #else
    - /*
    - * CPUID is a barrier to speculative execution.
    - * Prefetched instructions are automatically
    - * invalidated when modified.
    - */
    - asm volatile("cpuid"
    - : "=a" (tmp)
    - : "0" (1)
    - : "ebx", "ecx", "edx", "memory");
    + unsigned long tmp;
    +
    + asm volatile (
    + "movq %%ss, %0\n\t"
    + "pushq %0\n\t"
    + "pushq %%rsp\n\t"
    + "addq $8, (%%rsp)\n\t"
    + "pushfq\n\t"
    + "movq %%cs, %0\n\t"
    + "pushq %0\n\t"
    + "pushq $1f\n\t"
    + "iretq\n\t"
    + "1:"
    + : "=r" (tmp), "+r" (__sp) : : "cc", "memory");
    #endif
    }

    --
    2.9.3
    \
     
     \ /
      Last update: 2016-12-05 22:33    [W:4.984 / U:0.152 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site