Messages in this thread | | | Date | Fri, 25 Feb 2000 14:46:16 -0500 | From | Dave Anderson <> | Subject | [BUG] (alpha) kernel thread panics due to stale PTBR settings in 2.3.47 |
| |
<!doctype html public "-//w3c//dtd html 4.0 transitional//en"> <html> <tt>Hello,</tt><tt></tt> <p><tt> Sorry for the wide distribution -- I'm not sure who this should be directed</tt> <br><tt> to...</tt><tt></tt> <p><tt> We had been seeing panics in the alpha 2.3.41 stream where a kernel thread,</tt> <br><tt> typically one of the nfsd daemons or kswapd, fault on the swap_info swap_map</tt> <br><tt> address, which is a mapped (vmalloc'd) address. The problem was due to</tt> <br><tt> the disconnect between the active_mm pgd value and what's actually stored</tt> <br><tt> in the kernel task's ptbr value -- which is what gets loaded into the PTBR</tt> <br><tt> register with each alpha context switch. Eventually kernel tasks will find</tt> <br><tt> that the physical address stored in their thread_struct's ptbr become stale,</tt> <br><tt> as the page that they reference is freed and re-used elsewhere.</tt><tt></tt> <p><tt> I note that in 2.3.47, the problem looked to have been addressed by</tt> <br><tt> the addition of the enter_lazy_tlb() call in schedule():</tt><tt></tt> <p><tt> if (!mm) {</tt> <br><tt> if (next->active_mm) BUG();</tt> <br><tt> next->active_mm = oldmm;</tt> <br><tt> atomic_inc(&oldmm->mm_count);</tt> <br><tt> +++ enter_lazy_tlb(oldmm, next, this_cpu);</tt> <br><tt> }</tt><tt></tt> <p><tt> Unfortunately the alpha enter_lazy_tlb() doesn't do anything:</tt><tt></tt> <p><tt>static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk, unsigned cpu)</tt> <br><tt>{</tt> <br><tt>}</tt><tt></tt> <p><tt> If this is still a work in progress, excuse my interruption, but if not,</tt> <br><tt> the alpha enter_lazy_tlb() should update the kernel task's ptbr with the</tt> <br><tt> oldmm's pgd. Right?</tt><tt></tt> <p><tt> If you're interested in the details, here's the evidence from a 2.3.47 crash</tt> <br><tt> dump, in which kswapd panicked trying to reference a swap_map address at</tt> <br><tt> fffffe0000000032:</tt><tt></tt> <p><tt>crash> bt</tt> <br><tt>PID: 2 TASK: fffffc001fd64000 CPU: 0 COMMAND: "kswapd"</tt> <br><tt> #0 [fffffc001fd67ad0] crash_save_current_state at fffffc0000336ffc</tt> <br><tt> #1 [fffffc001fd67ae0] panic at fffffc00003271f8</tt> <br><tt> #2 [fffffc001fd67b80] die_if_kernel at fffffc00003113d0</tt> <br><tt> #3 [fffffc001fd67bb0] do_page_fault at fffffc000031fecc</tt> <br><tt> #4 [fffffc001fd67bf0] entMM at fffffc000031055c</tt> <br><tt> EFRAME: fffffc001fd67c28 R24: 0000000000000cec</tt> <br><tt> R0: 0000000000000001 R25: 0000000000000007</tt> <br><tt> R1: fffffe0000000032 R26: fffffc0000350aec <__delete_from_swap_cache+0x8c></tt> <br><tt> R2: 0000000000000003 R27: fffffc00003514c0</tt> <br><tt> R3: 0000190000000000 R28: 0000000000000000</tt> <br><tt> R4: fffffc000052d888 HAE: 0000000000000000</tt> <br><tt> R5: 0000000000000200 TRAP_A0: fffffe0000000032</tt> <br><tt> R6: fffffc00006329d0 TRAP_A1: 0000000000000001</tt> <br><tt> R7: fffffc001fd67dc0 TRAP_A2: 0000000000000000</tt> <br><tt> R8: fffffc001fd64000 PS: 0000000000000000</tt> <br><tt> R19: 0000000000000400 PC: fffffc0000351544 <__swap_free+0x84></tt> <br><tt> R20: fffffc00005317c0 GP: fffffc0000554030</tt> <br><tt> R21: 0000000000000000 R16: 0000190000000000</tt> <br><tt> R22: 0000000000000006 R17: 0000000000000001</tt> <br><tt> R23: fffffc0000345244 R18: 0000000000000059</tt> <br><tt> #5 [fffffc001fd67d10] __swap_free at fffffc0000351544</tt> <br><tt> #6 [fffffc001fd67d50] __delete_from_swap_cache at fffffc0000350aec</tt> <br><tt> #7 [fffffc001fd67d60] shrink_mmap at fffffc0000345460</tt> <br><tt> #8 [fffffc001fd67de0] do_try_to_free_pages at fffffc000034f87c</tt> <br><tt> #9 [fffffc001fd67e20] kswapd at fffffc000034fa2c</tt> <br><tt>#10 [fffffc001fd67e60] kernel_thread at fffffc00003107f0</tt><tt></tt> <p><tt> In the case above, the kswapd's ptbr references physical address</tt> <br><tt> 5bd8000, which has long since been freed and re-assigned to the</tt> <br><tt> kmem slab area:</tt><tt></tt> <p><tt>crash> task fffffc001fd64000 | grep ptbr</tt> <br><tt> ptbr = 0x2dec,</tt> <br><tt>crash> ptob 0x2dec</tt> <br><tt>2dec: 5bd8000</tt> <br><tt>crash> kmem -p 5bd8000</tt> <br><tt> PAGE PHYSICAL MAPPING INDEX CNT FLAGS</tt> <br><tt>fffffc0000c212e0 5bd8000 0000000000000000 106 1 uptodate,slab</tt><tt></tt> <p><tt> At the same time as the panic above, the 8 nfsd daemons and the two</tt> <br><tt> idle tasks *all* contained ptbr values referencing physical addresses that</tt> <br><tt> had been freed and re-used.</tt><tt></tt> <p><tt> Thanks,</tt> <br><tt> Dave Anderson</tt> <br><tt> anderson@missioncriticallinux.com</tt> <br> <br> <br> <br> </html>
| |