Messages in this thread | | | From | JeeHeng Sia <> | Subject | RE: Bug report: kernel paniced when system hibernates | Date | Thu, 18 May 2023 04:06:06 +0000 |
| |
> -----Original Message----- > From: Alexandre Ghiti <alexghiti@rivosinc.com> > Sent: Wednesday, May 17, 2023 4:33 PM > To: JeeHeng Sia <jeeheng.sia@starfivetech.com> > Cc: Song Shuai <suagrfillet@gmail.com>; robh@kernel.org; Andrew Jones <ajones@ventanamicro.com>; anup@brainfault.org; > palmer@rivosinc.com; Leyfoon Tan <leyfoon.tan@starfivetech.com>; Mason Huo <mason.huo@starfivetech.com>; Paul Walmsley > <paul.walmsley@sifive.com>; Conor Dooley <conor.dooley@microchip.com>; Guo Ren <guoren@kernel.org>; linux- > riscv@lists.infradead.org; linux-kernel@vger.kernel.org > Subject: Re: Bug report: kernel paniced when system hibernates > > On Tue, May 16, 2023 at 1:27 PM JeeHeng Sia > <jeeheng.sia@starfivetech.com> wrote: > > > > > > > > > -----Original Message----- > > > From: Alexandre Ghiti <alexghiti@rivosinc.com> > > > Sent: Tuesday, May 16, 2023 7:15 PM > > > To: JeeHeng Sia <jeeheng.sia@starfivetech.com> > > > Cc: Song Shuai <suagrfillet@gmail.com>; robh@kernel.org; Andrew Jones <ajones@ventanamicro.com>; anup@brainfault.org; > > > palmer@rivosinc.com; Leyfoon Tan <leyfoon.tan@starfivetech.com>; Mason Huo <mason.huo@starfivetech.com>; Paul Walmsley > > > <paul.walmsley@sifive.com>; Conor Dooley <conor.dooley@microchip.com>; Guo Ren <guoren@kernel.org>; linux- > > > riscv@lists.infradead.org; linux-kernel@vger.kernel.org > > > Subject: Re: Bug report: kernel paniced when system hibernates > > > > > > Hi JeeHeng, > > > > > > On Tue, May 16, 2023 at 11:55 AM JeeHeng Sia > > > <jeeheng.sia@starfivetech.com> wrote: > > > > > > > > Hi Song, > > > > > > > > Thanks for the investigation. Indeed, the exposure of the PMP reserved region to the kernel page table is causing the problem. > > > > Here is the similar report: https://groups.google.com/u/0/a/groups.riscv.org/g/sw-dev/c/ITXwaKfA6z8 > > > > > > IMO, we should discuss the kernel related stuff on the linux riscv ML, > > > I'm not subscribed to the group above and you did not answer my last > > > direct emails regarding this problem either. > > Hi Alex, it's strange that I haven't received a reply from you. > > I should stop using my personal email as it gets more and more > blocked, I don't know why. > > > Seems like I have forgot to update to the mailing list. > > By the way, the reason hibernation failed with the recent page table is because during the initiation of the hibernation process, the > hibernation core accesses the page table to check if the page can be saved. Since the page is reserved and is available to the page > table, the kernel tries to access the PMP region, which causes a kernel panic. The code can be found under: > > /kernel/power/snapshot.c/ line #1340 > > if (PageReserved(page) && (!kernel_page_present(page) || pfn_is_nosave(pfn))) > > Thanks for this pointer, it really helps! np. > > > If the virtual memory is mapped to the kernel page table, then hibernation core will try to access the page. > > > > > Thanks, > > > > > > Alex > > > > > > > > > > > Thanks > > > > Regards > > > > Jee Heng > > Is your issue also related to hibernation? My issue is related to memory testing/debugging. I am glad to see more and more people interested in the RISC-V Hibernation solution. > > > > > > > > > > -----Original Message----- > > > > > From: Song Shuai <suagrfillet@gmail.com> > > > > > Sent: Tuesday, May 16, 2023 5:24 PM > > > > > To: alexghiti@rivosinc.com; robh@kernel.org; Andrew Jones <ajones@ventanamicro.com>; anup@brainfault.org; > > > > > palmer@rivosinc.com; JeeHeng Sia <jeeheng.sia@starfivetech.com>; Leyfoon Tan <leyfoon.tan@starfivetech.com>; Mason > Huo > > > > > <mason.huo@starfivetech.com>; Paul Walmsley <paul.walmsley@sifive.com>; Conor Dooley <conor.dooley@microchip.com>; > > > Guo > > > > > Ren <guoren@kernel.org> > > > > > Cc: linux-riscv@lists.infradead.org; linux-kernel@vger.kernel.org > > > > > Subject: Bug report: kernel paniced when system hibernates > > > > > > > > > > Description of problem: > > > > > > > > > > The latest hibernation support[1] of RISC-V Linux produced a kernel panic. > > > > > The entire log has been posted at this link: https://termbin.com/sphl . > > > > > > > > > > How reproducible: > > > > > > > > > > You can reproduce it with the following step : > > > > > > > > > > 1. prepare the environment with > > > > > - Qemu-virt v8.0.0 (with OpenSbi v1.2) > > > > > - Linux v6.4-rc1 > > > > > > > > > > 2. start the Qemu virt > > > > > ```sh > > > > > $ cat ~/8_riscv/start_latest.sh > > > > > #!/bin/bash > > > > > /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \ > > > > > -smp 2 -m 4G -nographic -machine virt \ > > > > > -kernel /home/song/9_linux/linux/00_rv_test/arch/riscv/boot/Image \ > > > > > -append "root=/dev/vda ro eaylycon=uart8250,mmio,0x10000000 > > > > > early_ioremap_debug console=ttyS0 loglevel=8 memblock=debug > > > > > no_console_suspend audit=0 3" \ > > > > > -drive file=/home/song/8_riscv/fedora/stage4-disk.img,format=raw,id=hd0 \ > > > > > -device virtio-blk-device,drive=hd0 \ > > > > > -drive file=/home/song/8_riscv/fedora/adisk.qcow2,format=qcow2,id=hd1 \ > > > > > -device virtio-blk-device,drive=hd1 \ > > > > > -gdb tcp::1236 #-S > > > > > ``` > > > > > 3. execute hibernation > > > > > > > > > > ```sh > > > > > swapon /dev/vdb2 # this is my swap disk > > > > > > > > > > echo disk > /sys/power/state > > > > > ``` > > > > > > > > > > 4. Then you will encounter the kernel panic logged in the above link > > > > > > > > > > > > > > > Other Information: > > > > > > > > > > After my initial and incomplete dig-up, the commit (3335068f8721 > > > > > "riscv: Use PUD/P4D/PGD pages for the linear mapping")[2] > > > > > is closely related to this panic. This commit uses re-defined > > > > > `MIN_MEMBLOCK_ADDR` to discover the entire system memory > > > > > and extends the `va_pa_offset` from `kernel_map.phys_addr` to > > > > > `phys_ram_base` for linear memory mapping. > > > > > > > > > > If the firmware delivered the firmware memory region (like: a PMP > > > > > protected region in OpenSbi) without "no-map" propriety, > > > > > this commit will result in firmware memory being directly mapped by > > > > > `create_linear_mapping_page_table()`. > > > > > > > > > > We can see the mapping via ptdump : > > > > > ```c > > > > > ---[ Linear mapping ]--- > > > > > 0xff60000000000000-0xff60000000200000 0x0000000080000000 2M PMD D A G > > > > > . . W R V ------------- the firmware memory > > > > > 0xff60000000200000-0xff60000000c00000 0x0000000080200000 10M PMD D A G . . . R V > > > > > 0xff60000000c00000-0xff60000001000000 0x0000000080c00000 4M PMD D A G . . W R V > > > > > 0xff60000001000000-0xff60000001600000 0x0000000081000000 6M PMD D A G . . . R V > > > > > 0xff60000001600000-0xff60000040000000 0x0000000081600000 1002M PMD D A > > > > > G . . W R V > > > > > 0xff60000040000000-0xff60000100000000 0x00000000c0000000 3G PUD D A G . . W R V > > > > > ---[ Modules/BPF mapping ]--- > > > > > ---[ Kernel mapping ]--- > > > > > 0xffffffff80000000-0xffffffff80a00000 0x0000000080200000 10M PMD D A G . X . R V > > > > > 0xffffffff80a00000-0xffffffff80c00000 0x0000000080c00000 2M PMD D A G . . . R V > > > > > 0xffffffff80c00000-0xffffffff80e00000 0x0000000080e00000 2M PMD D A G . . W R V > > > > > 0xffffffff80e00000-0xffffffff81400000 0x0000000081000000 6M PMD D A G . . . R V > > > > > 0xffffffff81400000-0xffffffff81800000 0x0000000081600000 4M PMD > > > > > ``` > > > > > > > > > > In the hibernation process, `swsusp_save()` calls > > > > > `copy_data_pages(©_bm, &orig_bm)` to copy these two memory > > > > > bitmaps, > > > > > the Oops(load access fault) occurred while copying the page of > > > > > PAGE_OFFSET (which maps the firmware memory). > > > > > > > > > > I also did two other tests: > > > > > Test1: > > > > > > > > > > The hibernation works well in the kernel with the commit 3335068f8721 > > > > > reverted at least in the current environment. > > > > > > > > > > Test2: > > > > > > > > > > I built a simple kernel module to simulate the access of the value of > > > > > `PAGE_OFFSET` address, and the same panic occurred with the load > > > > > access fault. > > > > > So hibernation seems not the only case to trigger this panic. > > > > > > > > > > Finally, should we always leave the firmware memory with > > > > > `MEMBLOCK_NOMAP` flag by some efforts from Linux or OpenSbi (at least > > > > > in the current environment) or any other suggestions? > > > > > > > > > > Please correct me if I'm wrong. > > > > > > > > > > [1]: https://lore.kernel.org/r/20230330064321.1008373-5-jeeheng.sia@starfivetech.com > > > > > [2]: https://lore.kernel.org/r/20230324155421.271544-4-alexghiti@rivosinc.com > > > > > > > > > > -- > > > > > Thanks, > > > > > Song
| |