Messages in this thread | | | From | Ian Kumlien <> | Date | Mon, 16 Oct 2023 12:56:58 +0200 | Subject | Re: [bug] 6.5.7 - ixbe freezes and causes RCU deadlock? |
| |
And again, no oops visible this time
135476.059611] ixgbe 0000:07:00.0 eno3: Reset adapter [135483.747803] rcu: INFO: rcu_preempt self-detected stall on CPU [135483.753749] rcu: 3-....: (20999 ticks this GP) idle=ddf4/1/0x4000000000000000 softirq=997198/997198 fqs=3594 [135483.763852] rcu: (t=21015 jiffies g=4687825 q=371 ncpus=12) [135483.769694] rcu: rcu_preempt kthread timer wakeup didn't happen for 6637 jiffies! g4687825 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 [135483.781436] rcu: Possible timer handling issue on cpu=8 timer-softirq=960866 [135483.788752] rcu: rcu_preempt kthread starved for 6660 jiffies! g4687825 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=8 [135483.799540] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. [135483.808849] rcu: RCU grace-period kthread stack dump: [135483.814249] rcu: Stack dump where RCU GP kthread last ran: [135546.819253] rcu: INFO: rcu_preempt self-detected stall on CPU [135546.825177] rcu: 3-....: (83999 ticks this GP) idle=ddf4/1/0x4000000000000000 softirq=997198/997198 fqs=3594 [135546.835276] rcu: (t=84088 jiffies g=4687825 q=802 ncpus=12) [135546.841114] rcu: rcu_preempt kthread starved for 69713 jiffies! g4687825 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=8 [135546.851835] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. [135546.861177] rcu: RCU grace-period kthread stack dump: [135546.866576] rcu: Stack dump where RCU GP kthread last ran:
On Sun, Oct 15, 2023 at 2:01 PM Ian Kumlien <ian.kumlien@gmail.com> wrote: > > On Sun, Oct 15, 2023 at 5:38 AM Hillf Danton <hdanton@sina.com> wrote: > > > > On Sun, 15 Oct 2023 00:11:41 +0200 Ian Kumlien <ian.kumlien@gmail.com> > > > So, this keeps happening - it's happened for quite some time now... > > > I can't really reproduce it but it starts with a network adapter > > > freezing and ends with RCU errors > > > and watchdog reboot... :/ > > > > > > cat bug.txt | ./scripts/decode_stacktrace.sh vmlinux > > > [185433.169006] ------------[ cut here ]------------ > > > [185433.169018] NETDEV WATCHDOG: eno3 (ixgbe): transmit queue 2 timed out 9736 ms > > > [185433.169094] WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:525 > > > dev_watchdog (net/sched/sch_generic.c:525 (discriminator 3)) > > > > Watchdog reported eno3 tx hang. > > ... > > > > > > And in the IPMI console: > > > [185433.169621] ixgbe 0000:07:00.0 eno3: Reset adapter > > > [185444.166717] rcu: INFO: rcu_preempt self-detected stall on CPU > > > [185444.172665] rcu: 0-...!: (20999 ticks this GP) > > > idle=8d84/1/0x4000000000000000 softirq=1976223/1976223 fqs=2 > > > [185444.182681] rcu: (t=21015 jiffies g=6787421 q=738 ncpus=12) > > > [185444.188523] rcu: rcu_preempt kthread timer wakeup didn't happen > > > for 21009 jiffies! g6787421 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 > > > [185444.200361] rcu: Possible timer handling issue on cpu=8 timer-softirq=1196063 > > > > Timer on CPU8 is suspected to cause RCU stall. > > > > > [185444.207761] rcu: rcu_preempt kthread starved for 21032 jiffies! > > > g6787421 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=8 > > > [185444.218639] rcu: Unless rcu_preempt kthread gets sufficient CPU > > > time, OOM is now expected behavior. > > > [185444.227946] rcu: RCU grace-period kthread stack dump: > > > [185444.233347] rcu: Stack dump where RCU GP kthread last ran: > > > [185507.243156] rcu: INFO: rcu_preempt self-detected stall on CPU > > > [185507.249098] rcu: 0-....: (84002 ticks this GP) > > > idle=8d84/1/0x4000000000000000 softirq=1976223/1976223 fqs=1559 > > > [185507.259375] rcu: (t=84094 jiffies g=6787421 q=1213 ncpus=12) > > > [185570.265595] rcu: INFO: rcu_preempt self-detected stall on CPU > > > [185570.271532] rcu: 0-....: (147002 ticks this GP) > > > idle=8d84/1/0x4000000000000000 softirq=1976223/1976223 fqs=13844 > > > [185570.282016] rcu: (t=147117 jiffies g=6787421 q=1273 ncpus=12) > > > [185570.288049] rcu: rcu_preempt kthread timer wakeup didn't happen > > > for 13787 jiffies! g6787421 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 > > > [185570.299914] rcu: Possible timer handling issue on cpu=9 timer-softirq=1211534 > > > > Ditto on CPU9. > > > > No answer yet to why rcu stall was reported without any info about the timers > > on CPU8/9. > > Well... I can't really give you anymore information, all i can say is > that it leads to complete deadlock and eventual reboot by the hardware > watchdog...
| |