Messages in this thread | | | Date | Sat, 18 Dec 2021 18:39:46 +0800 | Subject | Re: [LKP] Re: [x86/mm/64] f154f29085: BUG:kernel_reboot-without-warning_in_boot_stage - clang KCOV? | From | Yin Fengwei <> |
| |
Hi Boris,
On 12/17/2021 8:52 PM, Borislav Petkov wrote: > Add Bruce and clang folks to the party. > > On Thu, Dec 16, 2021 at 08:21:15PM +0800, Yin Fengwei wrote: >> Hi Boris, >> >> On 12/16/2021 7:58 PM, Carel Si wrote: >>> Hi Boris, >>> >>> On Thu, Dec 16, 2021 at 11:06:59AM +0100, Borislav Petkov wrote: >>>> On Thu, Dec 16, 2021 at 03:04:16PM +0800, Yin Fengwei wrote: >>>>> The testing was with Qemu. >>>> >>>> This is hardly what I asked for. >>>> >>>>> And we found that the hang is related with clang-14. >>>> >>>> I saw that already. >>>> >>>>> The original report showed the kernel is built with clang-14: >>>>> # build kernel >>>>> cd linux >>>>> cp config-5.16.0-rc3-00003-gf154f290855b .config >>>>> make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules >>>>> make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install >>>> >>>> I saw that too. >>>> >>>>> Looks like KASAN related stub generated by clang-14 (KASAN_SHADOW_OFFSET and asan_report). >>>>> This function is early function called before kasan_init. >>>>> >>>>> Looks like we need to disable KASAN_SANITIZE for arch/x86/kernel/cpu/common.c. So clang-14 will >>>>> be happy with this kind of early TLB flush? Thanks. >>>> >>>> Ok, I don't understand: I asked for how exactly to reproduce and whether >>>> you can send me your vmlinux you built with your clang-14. What I get is >>>> some possible explanation about what might be happening. >>>> >>>> So what do you expect me to do? Say, "oh, sure, you're right, send me a >>>> patch" without even being able to see for myself what the root cause is? >>>> >>>> What if it is not the kernel's fault but clang-14 is miscompiling crap >>>> as in so many other cases? >>>> >>>> I built clang-14 and built with your .config and it works here fine. So >>>> why does yours fail? >>>> >>>> Or what's the point of all this? >> >> I had concern that our report is an invalid report because you can't reproduce >> it in your side. If that's the case, it could waste more your time. That's why >> I did check and shared what I got. I am very sorry if I did it wrong. > > Sure, you can always add your analysis but I'd like to reproduce myself > too. So, in the future, please answer the questions and then feel free > to add your analysis - I'll gladly have a look. Thanks a lot for sharing this. Lessons learnt here. Will follow this rule exactly in the future.
> > Which wasn't that far from the truth, btw. > > But it isn't KASAN but GCOV profiling. Or is it KCOV profiling which > clang does. > > That thing adds some counting glue to native_write_cr4(): > > (my comments from the actual singlestepping in qemu start with '##' below) > > movq $__llvm_gcov_ctr.48+8, %rbx ## mov $0xffffffff8837d3c0,%rbx > .LBB8_1: # %set_register > # =>This Inner Loop Header: Depth=1 > > jmp .Ltmp42 > ... > > .Ltmp42: # Block address taken > .LBB8_7: # %if.end79 > movq %rbx, %rax ## 0xffffffff8837d3c0 > shrq $3, %rax ## 0x1ffffffff106fa78 > movabsq $-2305847407260205056, %rcx # imm = 0xDFFFFC0000000000 ## 0xdffffc0000000000 > cmpb $0, (%rax,%rcx) > je .LBB8_9 > > so the memory address CMP accesses is something as nonsensical as > > 0xfffffbfff106fa78 > > so I'm guessing we need to setup something for that __llvm_gcov_ctr to > deref properly but I haven't dug deeper. > > The important thing is that this triggers with clang-13 and -14. gcc is > fine with the same config but that probably is because gcc does other > profiling - gcov - I guess. Looking at the resulting asm, it has a bunch > of those counter increments: > > incq __gcov0.native_write_cr4+88(%rip) # __gcov0.native_write_cr4[11] > > but no weird memory references. > > So, clang folks, what's up? > > The fix is simple but I'd like to understand first why does this fail > only with clang, 13 and newer. > > (I mean, melver pointed me to > > 380d53c45ff2 ("compiler_attributes.h: define __no_profile, add to noinstr") > > which explains why 13 and newer). > > Btw, joro, that second hunk is I think needed too because a couple of > lines earlier we set up the cr4 shadow so I think you should use it > instead of touching the hw CR4. > > --- > diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c > index 0083464de5e3..79b3d67addcc 100644 > --- a/arch/x86/kernel/cpu/common.c > +++ b/arch/x86/kernel/cpu/common.c > @@ -384,7 +384,7 @@ void native_write_cr0(unsigned long val) > } > EXPORT_SYMBOL(native_write_cr0); > > -void native_write_cr4(unsigned long val) > +void __no_profile native_write_cr4(unsigned long val) > { > unsigned long bits_changed = 0; > > diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c > index 75acb6027a87..68d2b7f9a913 100644 > --- a/arch/x86/kernel/head64.c > +++ b/arch/x86/kernel/head64.c > @@ -483,7 +483,7 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data) > /* Kill off the identity-map trampoline */ > reset_early_page_tables(); > > - __native_tlb_flush_global(native_read_cr4()); > + __native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4)); > > clear_bss(); > > > > Leaving in the rest for the newly added folks. I tried this fix and it works fine in my local env. Will test it also in our test box once we back to office. Thanks.
Regards Yin, Fengwei
> >> If you don't want to use lkp tool to reproduce the issue, following command >> could be used as well: >> >> Use Qemu command so only kernel image need be downloaded: >> qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G -s -S -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -nographic -append "console=ttyS0 earlyprintk=ttyS0,115200" >> to reproduce it. >> >> >> >> Regards >> Yin, Fengwei >> >> >> >>>> >>>> I mean, if you cannot send me what I ask for, you can say so. Then I can >>>> ignore this whole report altogether and waste my time somewhere else. >>> >>> We have uploaded vmlinuz, modules.cgz, config as well as other related file to: >>> https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/ >>> >>> Machine types can refer to: >>> https://zerobin.net/?e107cf7b56495d80#MQLh14wUT9Osv1tWCwiQx/okkAN48Nq+drVPE0PiNPw= >>> >>> If there's any other msg needed, pls feel free to propose, thanks. >>> >>> Below are our full steps to reproduce the issue: >>> >>> # download lkp-tests >>> $ git clone https://github.com/intel/lkp-tests.git >>> >>> $ cd lkp-tests/ >>> >>> # download vmlinuz >>> $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b >>> >>> # dowmload modules.cgz >>> $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/modules.cgz >>> >>> # download job-script which is attached >>> >>> # run lkp qemu >>> lkp-tests$ sudo bin/lkp qemu -k vmlinuz-5.16.0-rc3-00003-gf154f290855b -m modules.cgz job-script >>> >>> ~/lkp-tests/pkg/lkp-src ~/lkp-tests >>> x86_64 >>> ==> Making package: lkp-src 0-1 (Thu 16 Dec 2021 07:26:22 PM CST) >>> ==> Checking runtime dependencies... >>> ==> Checking buildtime dependencies... >>> ==> WARNING: Using existing $srcdir/ tree >>> ==> Removing existing $pkgdir/ directory... >>> ==> Starting build()... >>> make: Entering directory '/home/carel/lkp-tests/bin/event' >>> klcc -D_FORTIFY_SOURCE=2 -c -o wakeup.o wakeup.c >>> klcc -Wl,-O1,--sort-common,--as-needed,-z,relro -static -o wakeup wakeup.o >>> rm -f wakeup.o >>> strip wakeup >>> make: Leaving directory '/home/carel/lkp-tests/bin/event' >>> ==> Entering fakeroot environment... >>> x86_64 >>> ==> Starting package()... >>> ==> Creating package "lkp-src"... >>> 103987 blocks >>> renamed '/home/carel/.lkp/cache/lkp-x86_64.cgz.tmp' -> '/home/carel/.lkp/cache/lkp-x86_64.cgz' >>> ==> Leaving fakeroot environment. >>> ==> Finished making: lkp-src 0-1 (Thu 16 Dec 2021 07:26:24 PM CST) >>> ~/lkp-tests >>> 12 blocks >>> result_root: /home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0 >>> downloading initrds ... >>> use local modules: /home/carel/.lkp/cache/modules.cgz >>> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/debian/debian-10.4-x86_64-20200603.cgz -N -P /home/carel/.lkp/cache/osimage/debian >>> 440459 blocks >>> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/run-ipconfig_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz >>> 1773 blocks >>> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/lkp_20210707.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz >>> 2321 blocks >>> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/rsync-rootfs_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz >>> 6856 blocks >>> exec command: qemu-system-x86_64 -enable-kvm -fsdev local,id=test_dev,path=/home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0,security_model=none -device virtio-9p-pci,fsdev=test_dev,mount_tag=9p/virtfs_mount -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -append root=/dev/ram0 user=lkp job=/lkp/jobs/scheduled/vm-snb-192/boot-1-debian-10.4-x86_64-20200603.cgz-f154f290855b070cc94dd44ad253c0ef8a9337bb-20211208-23538-lnvkeg-5.yaml ARCH=x86_64 kconfig=x86_64-randconfig-a013-20211207 branch=tip/x86/mm commit=f154f290855b070cc94dd44ad253c0ef8a9337bb BOOT_IMAGE=/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b vmalloc=128M initramfs_async=0 page_owner=on max_uptime=600 RESULT_ROOT=/result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/3 LKP_LOCAL_RUN=1 selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw ip=dhcp result_service=9p/virtfs_mount -initrd /home/carel/.lkp/cache/final_initrd -smp 2 -m 3144M -no-reboot -watchdog i6300esb -rtc base=localtime -device e1000,netdev=net0 -netdev user,id=net0 -display none -monitor null -serial stdio >>> early console in setup code >>> early console in extract_kernel >>> input_data: 0x0000000006ffc2e0 >>> input_len: 0x000000000260cb2b >>> output: 0x0000000001000000 >>> output_len: 0x00000000079e7da4 >>> kernel_total_size: 0x0000000008a2c000 >>> needed_size: 0x0000000008c00000 >>> trampoline_32bit: 0x000000000009d000 >>> Physical KASLR using RDTSC... >>> Virtual KASLR using RDTSC... >>> >>> Decompressing Linux... Parsing ELF... Performing relocations... done. >>> Booting the kernel. >>> >>>> >>>> -- >>>> Regards/Gruss, >>>> Boris. >>>> >>>> SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg >
| |