lkml.org 
[lkml]   [2013]   [Mar]   [4]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: BUG: IPv4: Attempt to release TCP socket in state 1
(Cc'ing the right netdev mailing list...)

On 03/05/2013 08:01 AM, dormando wrote:
> Hi!
>
> I have a (core lockup?) with 3.7.6+ and 3.8.2 which appears to be under
> ixgbe. The machine appears to still be up but network stays in a severely
> hobbled state. Either lagging or not responding to the network at all.
>
> On a new box the hang happens within 8-24 hours of giving it production
> network traffic. On an older machine (6 cores instead of 8, etc) it can
> run for a week or more before hanging.
>
> The hang from 3.7 might be slightly different than 3.8. They seem to be
> mostly the same aside from 3.8 hanging in the GRO path. Don't see anything
> obvious in 3.9-rc1 that would fix it, and haven't tried 3.9-rc1.
>
> I've not yet figured out how to reproduce outside of production (as
> always, sigh). This doesn't seem to happen with 3.6.6, but we have
> different and less frequent kernel panics there.
>
> From 3.7:
>
> [21934.669780] IPv4: Attempt to release TCP socket in state 1
> ffff882785e3db00
> [21969.265883] ------------[ cut here ]------------
> [21969.265898] WARNING: at net/sched/sch_generic.c:255
> dev_watchdog+0x258/0x270()
> [21969.265900] Hardware name: X9DR3-F
> [21969.265902] NETDEV WATCHDOG: eth2 (ixgbe): transmit queue 11 timed out
> [21969.265903] Modules linked in: macvlan bridge ipmi_watchdog
> ipmi_devintf coretemp ghash_clmulni_intel gpio_ich microcode ixgbe sb_edac
> mdio lpc_ich edac_core mei mfd_core ipmi_si ipmi_msghandler isci libsas
> igb
> [21969.265930] Pid: 0, comm: swapper/10 Not tainted 3.7.8 #1
> [21969.265931] Call Trace:
> [21969.265933] <IRQ> [<ffffffff810484ff>] warn_slowpath_common+0x7f/0xc0
> [21969.265945] [<ffffffff815a712e>] ? ip_local_deliver_finish+0xde/0x290
> [21969.265948] [<ffffffff810485f6>] warn_slowpath_fmt+0x46/0x50
> [21969.265950] [<ffffffff815a69b9>] ? ip_rcv_finish+0x119/0x360
> [21969.265953] [<ffffffff8157d538>] dev_watchdog+0x258/0x270
> [21969.265956] [<ffffffff8157d2e0>] ? __netdev_watchdog_up+0x80/0x80
> [21969.265960] [<ffffffff81058349>] call_timer_fn+0x49/0x130
> [21969.265963] [<ffffffff81078f9f>] ? scheduler_tick+0x15f/0x190
> [21969.265965] [<ffffffff81058944>] run_timer_softirq+0x224/0x290
> [21969.265967] [<ffffffff81058066>] ? update_process_times+0x76/0x90
> [21969.265969] [<ffffffff8157d2e0>] ? __netdev_watchdog_up+0x80/0x80
> [21969.265974] [<ffffffff8108b4f4>] ? ktime_get+0x54/0xe0
> [21969.265977] [<ffffffff810509c7>] __do_softirq+0xc7/0x230
> [21969.265990] [<ffffffff816757cc>] call_softirq+0x1c/0x30
> [21969.265995] [<ffffffff81004475>] do_softirq+0x55/0x90
> [21969.265997] [<ffffffff810507c5>] irq_exit+0x85/0xa0
> [21969.265999] [<ffffffff81675dfe>] smp_apic_timer_interrupt+0x6e/0x99
> [21969.266002] [<ffffffff816751ca>] apic_timer_interrupt+0x6a/0x70
> [21969.266003] <EOI> [<ffffffff8166b17a>] ? __schedule+0x3aa/0x750
> [21969.266011] [<ffffffff8100b2ed>] ? mwait_idle+0xad/0x1f0
> [21969.266013] [<ffffffff8100a7a3>] cpu_idle+0xb3/0x100
> [21969.266017] [<ffffffff816632e3>] start_secondary+0x1c9/0x1d0
> [21969.266019] ---[ end trace 0739ad788910e77e ]---
> [21969.266059] ixgbe 0000:83:00.0 eth2: Reset adapter
> [22019.676899] INFO: rcu_sched self-detected stall on CPU { 30} (t=15001
> jiffies)
> [22019.676963] Pid: 0, comm: swapper/30 Tainted: G W 3.7.8 #1
> [22019.676966] Call Trace:
> [22019.676968] <IRQ> [<ffffffff810bb144>]
> rcu_check_callbacks+0x1b4/0x600
> [22019.676985] [<ffffffff8107e1b8>] ? account_system_time+0xe8/0x1e0
> [22019.676988] [<ffffffff81058038>] update_process_times+0x48/0x90
> [22019.676993] [<ffffffff81092aa7>] tick_sched_timer+0x77/0x160
> [22019.677006] [<ffffffff8106f66d>] __run_hrtimer+0x7d/0x1c0
> [22019.677008] [<ffffffff81092a30>] ? tick_setup_sched_timer+0x110/0x110
> [22019.677010] [<ffffffff8106fa26>] hrtimer_interrupt+0xf6/0x230
> [22019.677015] [<ffffffff81675df9>] smp_apic_timer_interrupt+0x69/0x99
> [22019.677018] [<ffffffff816751ca>] apic_timer_interrupt+0x6a/0x70
> [22019.677023] [<ffffffff815afaa0>] ?
> __inet_lookup_established+0xc0/0x280
> [22019.677026] [<ffffffff815a68a0>] ? inet_del_protocol+0x40/0x40
> [22019.677030] [<ffffffff815cc383>] tcp_v4_early_demux+0xa3/0x170
> [22019.677033] [<ffffffff815a69ed>] ip_rcv_finish+0x14d/0x360
> [22019.677035] [<ffffffff815a6f66>] ip_rcv+0x226/0x310
> [22019.677041] [<ffffffff815609f2>] __netif_receive_skb+0x492/0x640
> [22019.677043] [<ffffffff81074209>] ? __wake_up_common+0x59/0x90
> [22019.677051] [<ffffffffa00f284b>] ? ixgbe_poll+0xe3b/0x1140 [ixgbe]
> [22019.677054] [<ffffffff81560c94>] process_backlog+0xf4/0x1e0
> [22019.677056] [<ffffffff815619c5>] net_rx_action+0xf5/0x260
> [22019.677070] [<ffffffff810509c7>] __do_softirq+0xc7/0x230
> [22019.677072] [<ffffffff816757cc>] call_softirq+0x1c/0x30
> [22019.677076] [<ffffffff81004475>] do_softirq+0x55/0x90
> [22019.677078] [<ffffffff810507c5>] irq_exit+0x85/0xa0
> [22019.677080] [<ffffffff81675d16>] do_IRQ+0x66/0xe0
> [22019.677084] [<ffffffff8166c8aa>] common_interrupt+0x6a/0x6a
> [22019.677085] <EOI> [<ffffffff8166b17a>] ? __schedule+0x3aa/0x750
> [22019.677090] [<ffffffff8100b2ed>] ? mwait_idle+0xad/0x1f0
> [22019.677092] [<ffffffff8100a7a3>] cpu_idle+0xb3/0x100
> [22019.677096] [<ffffffff816632e3>] start_secondary+0x1c9/0x1d0
> [22188.695704] INFO: task kworker/10:2:676 blocked for more than 120
> seconds.
> [22188.695750] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [22188.695807] kworker/10:2 D ffffffff81806e40 0 676 2
> 0x00000000
> [22188.695813] ffff882ff9dadad8 0000000000000046 ffff882ff9082d80
> 00000000000126c0
> [22188.695816] ffff882ff9dadfd8 ffff882ff9dac010 00000000000126c0
> 00000000000126c0
> [22188.695818] ffff882ff9dadfd8 00000000000126c0 ffff882ffb185b00
> ffff882ff9082d80
> [22188.695820] Call Trace:
> [22188.695830] [<ffffffff8166b5e9>] schedule+0x29/0x70
> [22188.695833] [<ffffffff816698e5>] schedule_timeout+0x165/0x200
> [22188.695838] [<ffffffff810796b5>] ? ttwu_do_wakeup+0x45/0x100
> [22188.695840] [<ffffffff810797b9>] ? T.1871+0x49/0x60
> [22188.695843] [<ffffffff8107c28e>] ? try_to_wake_up+0x23e/0x2b0
> [22188.695845] [<ffffffff8166ac58>] wait_for_common+0xc8/0x160
> [22188.695847] [<ffffffff8107c300>] ? try_to_wake_up+0x2b0/0x2b0
> [22188.695852] [<ffffffff810b90c0>] ? rcu_cpu_stall_reset+0x60/0x60
> [22188.695854] [<ffffffff8166adcd>] wait_for_completion+0x1d/0x20
> [22188.695859] [<ffffffff810aed96>] __stop_cpus+0x56/0x80
> [22188.695861] [<ffffffff810b90c0>] ? rcu_cpu_stall_reset+0x60/0x60
> [22188.695864] [<ffffffff810aee0d>] try_stop_cpus+0x4d/0x80
> [22188.695867] [<ffffffff810bb62a>]
> synchronize_sched_expedited+0x9a/0x120
> [22188.695869] [<ffffffff810bb6be>] synchronize_rcu_expedited+0xe/0x10
> [22188.695874] [<ffffffff8155a8e5>] synchronize_net+0x25/0x30
> [22188.695880] [<ffffffff8157dbb4>] dev_deactivate_many+0x254/0x260
> [22188.695882] [<ffffffff8157dbed>] dev_deactivate+0x2d/0x40
> [22188.695886] [<ffffffff8156fff4>] linkwatch_do_dev+0x34/0x60
> [22188.695888] [<ffffffff815701d3>] __linkwatch_run_queue+0xf3/0x1e0
> [22188.695891] [<ffffffff815702e5>] linkwatch_event+0x25/0x30
> [22188.695894] [<ffffffff81064180>] process_one_work+0x160/0x460
> [22188.695896] [<ffffffff815702c0>] ? __linkwatch_run_queue+0x1e0/0x1e0
> [22188.695899] [<ffffffff8106631b>] worker_thread+0x12b/0x3d0
> [22188.695901] [<ffffffff810661f0>] ? manage_workers+0x300/0x300
> [22188.695904] [<ffffffff8106b26e>] kthread+0xce/0xe0
> [22188.695907] [<ffffffff8106b1a0>] ?
> kthread_freezable_should_stop+0x70/0x70
> [22188.695911] [<ffffffff8167475c>] ret_from_fork+0x7c/0xb0
> [22188.695913] [<ffffffff8106b1a0>] ?
> kthread_freezable_should_stop+0x70/0x70
>
> [tons of processes hung in a similar way]
>
> Then every few hundred seconds swapper bails:
>
> [22919.239167] INFO: rcu_sched self-detected stall on CPU { 30} (t=240021
> jiffies)
> [22919.239409] Pid: 0, comm: swapper/30 Tainted: G W 3.7.8 #1
> [22919.239411] Call Trace:
> [22919.239413] <IRQ> [<ffffffff810bb144>]
> rcu_check_callbacks+0x1b4/0x600
> [22919.239430] [<ffffffff8107e1b8>] ? account_system_time+0xe8/0x1e0
> [22919.239434] [<ffffffff81058038>] update_process_times+0x48/0x90
> [22919.239439] [<ffffffff81092aa7>] tick_sched_timer+0x77/0x160
> [22919.239442] [<ffffffff8106f66d>] __run_hrtimer+0x7d/0x1c0
> [22919.239445] [<ffffffff81092a30>] ? tick_setup_sched_timer+0x110/0x110
> [22919.239447] [<ffffffff8106fa26>] hrtimer_interrupt+0xf6/0x230
> [22919.239453] [<ffffffff81675df9>] smp_apic_timer_interrupt+0x69/0x99
> [22919.239455] [<ffffffff816751ca>] apic_timer_interrupt+0x6a/0x70
> [22919.239461] [<ffffffff815afaab>] ?
> __inet_lookup_established+0xcb/0x280
> [22919.239463] [<ffffffff815a68a0>] ? inet_del_protocol+0x40/0x40
> [22919.239468] [<ffffffff815cc383>] tcp_v4_early_demux+0xa3/0x170
> [22919.239470] [<ffffffff815a69ed>] ip_rcv_finish+0x14d/0x360
> [22919.239472] [<ffffffff815a6f66>] ip_rcv+0x226/0x310
> [22919.239478] [<ffffffff815609f2>] __netif_receive_skb+0x492/0x640
> [22919.239481] [<ffffffff81074209>] ? __wake_up_common+0x59/0x90
> [22919.239490] [<ffffffffa00f284b>] ? ixgbe_poll+0xe3b/0x1140 [ixgbe]
> [22919.239493] [<ffffffff81560c94>] process_backlog+0xf4/0x1e0
> [22919.239495] [<ffffffff815619c5>] net_rx_action+0xf5/0x260
> [22919.239499] [<ffffffff810509c7>] __do_softirq+0xc7/0x230
> [22919.239501] [<ffffffff816757cc>] call_softirq+0x1c/0x30
> [22919.239505] [<ffffffff81004475>] do_softirq+0x55/0x90
> [22919.239507] [<ffffffff810507c5>] irq_exit+0x85/0xa0
> [22919.239509] [<ffffffff81675d16>] do_IRQ+0x66/0xe0
> [22919.239513] [<ffffffff8166c8aa>] common_interrupt+0x6a/0x6a
> [22919.239514] <EOI> [<ffffffff8166b17a>] ? __schedule+0x3aa/0x750
> [22919.239520] [<ffffffff8100b2ed>] ? mwait_idle+0xad/0x1f0
> [22919.239522] [<ffffffff8100a7a3>] cpu_idle+0xb3/0x100
> [22919.239526] [<ffffffff816632e3>] start_secondary+0x1c9/0x1d0
> [23099.151590] INFO: rcu_sched self-detected stall on CPU { 30} (t=285025
> jiffies)
> [23099.151823] Pid: 0, comm: swapper/30 Tainted: G W
> [23099.151825] Call Trace:
> [23099.151827] <IRQ> [<ffffffff810bb144>]
> rcu_check_callbacks+0x1b4/0x600
> [23099.151841] [<ffffffff8107e1b8>] ? account_system_time+0xe8/0x1e0
> [23099.151845] [<ffffffff81058038>] update_process_times+0x48/0x90
> [23099.151849] [<ffffffff81092aa7>] tick_sched_timer+0x77/0x160
> [23099.151853] [<ffffffff8106f66d>] __run_hrtimer+0x7d/0x1c0
> [23099.151856] [<ffffffff81092a30>] ? tick_setup_sched_timer+0x110/0x110
> [23099.151857] [<ffffffff8106fa26>] hrtimer_interrupt+0xf6/0x230
> [23099.151863] [<ffffffff81675df9>] smp_apic_timer_interrupt+0x69/0x99
> [23099.151865] [<ffffffff816751ca>] apic_timer_interrupt+0x6a/0x70
> [23099.151870] [<ffffffff815afb53>] ?
> __inet_lookup_established+0x173/0x280
> [23099.151873] [<ffffffff815a68a0>] ? inet_del_protocol+0x40/0x40
> [23099.151877] [<ffffffff815cc383>] tcp_v4_early_demux+0xa3/0x170
> [23099.151880] [<ffffffff815a69ed>] ip_rcv_finish+0x14d/0x360
> [23099.151882] [<ffffffff815a6f66>] ip_rcv+0x226/0x310
> [23099.151887] [<ffffffff815609f2>] __netif_receive_skb+0x492/0x640
> [23099.151890] [<ffffffff81074209>] ? __wake_up_common+0x59/0x90
> [23099.151897] [<ffffffffa00f284b>] ? ixgbe_poll+0xe3b/0x1140 [ixgbe]
> [23099.151900] [<ffffffff81560c94>] process_backlog+0xf4/0x1e0
> [23099.151902] [<ffffffff815619c5>] net_rx_action+0xf5/0x260
> [23099.151906] [<ffffffff810509c7>] __do_softirq+0xc7/0x230
> [23099.151908] [<ffffffff816757cc>] call_softirq+0x1c/0x30
> [23099.151912] [<ffffffff81004475>] do_softirq+0x55/0x90
> [23099.151914] [<ffffffff810507c5>] irq_exit+0x85/0xa0
> [23099.151916] [<ffffffff81675d16>] do_IRQ+0x66/0xe0
> [23099.151920] [<ffffffff8166c8aa>] common_interrupt+0x6a/0x6a
> [23099.151920] <EOI> [<ffffffff8166b17a>] ? __schedule+0x3aa/0x750
> [23099.151926] [<ffffffff8100b2ed>] ? mwait_idle+0xad/0x1f0
> [23099.151928] [<ffffffff8100a7a3>] cpu_idle+0xb3/0x100
> [23099.151931] [<ffffffff816632e3>] start_secondary+0x1c9/0x1d0
>
> Under 3.8.2:
>
> [33486.326977] IPv4: Attempt to release TCP socket in state 1
> ffff883269ea2300
> [33486.342971] IPv4: Attempt to release TCP socket in state 1
> ffff8835efccbf00
> [33505.595925] ------------[ cut here ]------------
> [33505.595934] WARNING: at net/sched/sch_generic.c:254
> dev_watchdog+0x258/0x270()
> [33505.595935] Hardware name: X9DR3-F
> [33505.595937] NETDEV WATCHDOG: eth2 (ixgbe): transmit queue 0 timed out
> [33505.595938] Modules linked in: macvlan iptable_nat nf_nat_ipv4 nf_nat
> bridge coretemp ghash_clmulni_intel gpio_ich ixgbe microcode sb_edac mei
> lpc_ich edac_core mfd_core mdio isci libsas igb ptp pps_core
> [33505.595951] Pid: 0, comm: swapper/4 Not tainted 3.8.2 #2
> [33505.595952] Call Trace:
> [33505.595954] <IRQ> [<ffffffff8104964f>] warn_slowpath_common+0x7f/0xc0
> [33505.595960] [<ffffffff81049746>] warn_slowpath_fmt+0x46/0x50
> [33505.595962] [<ffffffff815a1548>] dev_watchdog+0x258/0x270
> [33505.595965] [<ffffffff815a12f0>] ? __netdev_watchdog_up+0x80/0x80
> [33505.595968] [<ffffffff81059259>] call_timer_fn+0x49/0x130
> [33505.595972] [<ffffffff8107a07f>] ? scheduler_tick+0x15f/0x190
> [33505.595974] [<ffffffff81059854>] run_timer_softirq+0x224/0x290
> [33505.595976] [<ffffffff81058f76>] ? update_process_times+0x76/0x90
> [33505.595978] [<ffffffff815a12f0>] ? __netdev_watchdog_up+0x80/0x80
> [33505.595981] [<ffffffff8108ebd4>] ? ktime_get+0x54/0xe0
> [33505.595983] [<ffffffff810518a7>] __do_softirq+0xc7/0x230
> [33505.595987] [<ffffffff8168fe0c>] call_softirq+0x1c/0x30
> [33505.595990] [<ffffffff81004415>] do_softirq+0x55/0x90
> [33505.595993] [<ffffffff810516a5>] irq_exit+0x85/0xa0
> [33505.595996] [<ffffffff8169042e>] smp_apic_timer_interrupt+0x6e/0x99
> [33505.596000] [<ffffffff8168f80a>] apic_timer_interrupt+0x6a/0x70
> [33505.596002] <EOI> [<ffffffff8168567c>] ? __schedule+0x3ac/0x750
> [33505.596009] [<ffffffff8100b1fd>] ? mwait_idle+0xad/0x1f0
> [33505.596011] [<ffffffff8100a743>] cpu_idle+0xb3/0x100
> [33505.596014] [<ffffffff8167d7d2>] start_secondary+0x1d7/0x1de
> [33505.596015] ---[ end trace 3d817d7c7ae67386 ]---
> [33505.596064] ixgbe 0000:83:00.0 eth2: Reset adapter
> [33556.011932] INFO: rcu_sched self-detected stall on CPU { 24} (t=15001
> jiffies g=1985385 c=1985384 q=270786)
> [33556.011968] Pid: 0, comm: swapper/24 Tainted: G W 3.8.2 #2
> [33556.011970] Call Trace:
> [33556.011972] <IRQ> [<ffffffff810bea1e>]
> rcu_check_callbacks+0x21e/0x7c0
> [33556.011986] [<ffffffff8107f518>] ? account_system_time+0xe8/0x1e0
> [33556.011992] [<ffffffff81058f48>] update_process_times+0x48/0x90
> [33556.011996] [<ffffffff81095e06>] tick_sched_timer+0x56/0x130
> [33556.012000] [<ffffffff8107099d>] __run_hrtimer+0x7d/0x1c0
> [33556.012002] [<ffffffff81095db0>] ? tick_setup_sched_timer+0x110/0x110
> [33556.012004] [<ffffffff81070d56>] hrtimer_interrupt+0xf6/0x230
> [33556.012010] [<ffffffff81690429>] smp_apic_timer_interrupt+0x69/0x99
> [33556.012013] [<ffffffff8168f80a>] apic_timer_interrupt+0x6a/0x70
> [33556.012017] [<ffffffff815d3deb>] ?
> __inet_lookup_established+0xcb/0x2d0
> [33556.012020] [<ffffffff815cab80>] ? inet_del_protocol+0x40/0x40
> [33556.012024] [<ffffffff815f078c>] tcp_v4_early_demux+0xac/0x170
> [33556.012025] [<ffffffff815caccd>] ip_rcv_finish+0x14d/0x360
> [33556.012027] [<ffffffff815cb246>] ip_rcv+0x226/0x310
> [33556.012032] [<ffffffff815841a2>] __netif_receive_skb+0x492/0x640
> [33556.012034] [<ffffffff8158455d>] netif_receive_skb+0x2d/0x90
> [33556.012036] [<ffffffff815ed450>] ? tcp4_gro_receive+0xb0/0x130
> [33556.012038] [<ffffffff81584655>] napi_gro_complete+0x95/0xe0
> [33556.012040] [<ffffffff81584956>] dev_gro_receive+0x2b6/0x3b0
> [33556.012043] [<ffffffff8158508b>] napi_gro_receive+0x5b/0x130
> [33556.012051] [<ffffffffa01db04a>] ixgbe_poll+0x54a/0x1180 [ixgbe]
> [33556.012054] [<ffffffff810792fa>] ? enqueue_task+0x6a/0x80
> [33556.012056] [<ffffffff81584c15>] net_rx_action+0xf5/0x260
> [33556.012058] [<ffffffff810518a7>] __do_softirq+0xc7/0x230
> [33556.012061] [<ffffffff8168fe0c>] call_softirq+0x1c/0x30
> [33556.012064] [<ffffffff81004415>] do_softirq+0x55/0x90
> [33556.012066] [<ffffffff810516a5>] irq_exit+0x85/0xa0
> [33556.012068] [<ffffffff81690346>] do_IRQ+0x66/0xe0
> [33556.012071] [<ffffffff81686daa>] common_interrupt+0x6a/0x6a
> [33556.012073] <EOI> [<ffffffff8168567c>] ? __schedule+0x3ac/0x750
> [33556.012078] [<ffffffff8100b1fd>] ? mwait_idle+0xad/0x1f0
> [33556.012080] [<ffffffff8100a743>] cpu_idle+0xb3/0x100
> [33556.012082] [<ffffffff8167d7d2>] start_secondary+0x1d7/0x1de
> [33716.090584] INFO: task kworker/4:2:882 blocked for more than 120
> seconds.
> [33716.090602] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [33716.090618] kworker/4:2 D ffffffff81807160 0 882 2
> 0x00000000
> [33716.090622] ffff881fd2547ad8 0000000000000046 ffff881fd0ac2dc0
> 0000000000012700
> [33716.090624] ffff881fd2547fd8 ffff881fd2546010 0000000000012700
> 0000000000012700
> [33716.090626] ffff881fd2547fd8 0000000000012700 ffff881fd3655b80
> ffff881fd0ac2dc0
> [33716.090628] Call Trace:
> [33716.090639] [<ffffffff81685ae9>] schedule+0x29/0x70
> [33716.090642] [<ffffffff81683de5>] schedule_timeout+0x165/0x200
> [33716.090647] [<ffffffff810283fe>] ? physflat_send_IPI_mask+0xe/0x10
> [33716.090650] [<ffffffff8107d02e>] ? try_to_wake_up+0x23e/0x2b0
> [33716.090653] [<ffffffff81685158>] wait_for_common+0xc8/0x160
> [33716.090654] [<ffffffff8107d0a0>] ? try_to_wake_up+0x2b0/0x2b0
> [33716.090660] [<ffffffff810bc890>] ? rcu_cpu_stall_reset+0x60/0x60
> [33716.090662] [<ffffffff816852cd>] wait_for_completion+0x1d/0x20
> [33716.090665] [<ffffffff810b2536>] __stop_cpus+0x56/0x80
> [33716.090667] [<ffffffff810bc890>] ? rcu_cpu_stall_reset+0x60/0x60
> [33716.090669] [<ffffffff810b25ad>] try_stop_cpus+0x4d/0x80
> [33716.090672] [<ffffffff810bf0bb>]
> synchronize_sched_expedited+0xfb/0x1d0
> [33716.090674] [<ffffffff810bf19e>] synchronize_rcu_expedited+0xe/0x10
> [33716.090678] [<ffffffff8157e1f5>] synchronize_net+0x25/0x30
> [33716.090683] [<ffffffff815a1bc4>] dev_deactivate_many+0x254/0x260
> [33716.090685] [<ffffffff815a1bfd>] dev_deactivate+0x2d/0x40
> [33716.090688] [<ffffffff81593dc4>] linkwatch_do_dev+0x34/0x60
> [33716.090690] [<ffffffff81593fa3>] __linkwatch_run_queue+0xf3/0x1e0
> [33716.090692] [<ffffffff815940b5>] linkwatch_event+0x25/0x30
> [33716.090696] [<ffffffff810653f8>] process_one_work+0x168/0x450
> [33716.090699] [<ffffffff8106757b>] worker_thread+0x12b/0x3d0
> [33716.090702] [<ffffffff81067450>] ? manage_workers+0x300/0x300
> [33716.090704] [<ffffffff8106c5ee>] kthread+0xce/0xe0
> [33716.090706] [<ffffffff8106c520>] ?
> kthread_freezable_should_stop+0x70/0x70
> [33716.090709] [<ffffffff8168ec5c>] ret_from_fork+0x7c/0xb0
> [33716.090711] [<ffffffff8106c520>] ?
> kthread_freezable_should_stop+0x70/0x70
>
> [more hung processes bailing]
>
> [37335.739761] INFO: rcu_sched self-detected stall on CPU { 24} (t=960083
> jiffies g=1985385 c=1985384 q=19390495)
> [37335.739828] Pid: 0, comm: swapper/24 Tainted: G W 3.8.2 #2
> [37335.739830] Call Trace:
> [37335.739832] <IRQ> [<ffffffff810bea1e>]
> rcu_check_callbacks+0x21e/0x7c0
> [37335.739847] [<ffffffff8107f518>] ? account_system_time+0xe8/0x1e0
> [37335.739853] [<ffffffff81058f48>] update_process_times+0x48/0x90
> [37335.739857] [<ffffffff81095e06>] tick_sched_timer+0x56/0x130
> [37335.739860] [<ffffffff8107099d>] __run_hrtimer+0x7d/0x1c0
> [37335.739863] [<ffffffff81095db0>] ? tick_setup_sched_timer+0x110/0x110
> [37335.739865] [<ffffffff81070d56>] hrtimer_interrupt+0xf6/0x230
> [37335.739871] [<ffffffff81690429>] smp_apic_timer_interrupt+0x69/0x99
> [37335.739874] [<ffffffff8168f80a>] apic_timer_interrupt+0x6a/0x70
> [37335.739878] [<ffffffff815d3def>] ?
> __inet_lookup_established+0xcf/0x2d0
> [37335.739880] [<ffffffff815cab80>] ? inet_del_protocol+0x40/0x40
> [37335.739884] [<ffffffff815f078c>] tcp_v4_early_demux+0xac/0x170
> [37335.739886] [<ffffffff815caccd>] ip_rcv_finish+0x14d/0x360
> [37335.739888] [<ffffffff815cb246>] ip_rcv+0x226/0x310
> [37335.739892] [<ffffffff815841a2>] __netif_receive_skb+0x492/0x640
> [37335.739895] [<ffffffff8158455d>] netif_receive_skb+0x2d/0x90
> [37335.739897] [<ffffffff815ed450>] ? tcp4_gro_receive+0xb0/0x130
> [37335.739899] [<ffffffff81584655>] napi_gro_complete+0x95/0xe0
> [37335.739901] [<ffffffff81584956>] dev_gro_receive+0x2b6/0x3b0
> [37335.739903] [<ffffffff8158508b>] napi_gro_receive+0x5b/0x130
> [37335.739911] [<ffffffffa01db04a>] ixgbe_poll+0x54a/0x1180 [ixgbe]
> [37335.739915] [<ffffffff810792fa>] ? enqueue_task+0x6a/0x80
> [37335.739917] [<ffffffff81584c15>] net_rx_action+0xf5/0x260
> [37335.739919] [<ffffffff810518a7>] __do_softirq+0xc7/0x230
> [37335.739922] [<ffffffff8168fe0c>] call_softirq+0x1c/0x30
> [37335.739927] [<ffffffff81004415>] do_softirq+0x55/0x90
> [37335.739928] [<ffffffff810516a5>] irq_exit+0x85/0xa0
> [37335.739931] [<ffffffff81690346>] do_IRQ+0x66/0xe0
> [37335.739937] [<ffffffff81686daa>] common_interrupt+0x6a/0x6a
> [37335.739938] <EOI> [<ffffffff8168567c>] ? __schedule+0x3ac/0x750
> [37335.739943] [<ffffffff8100b1fd>] ? mwait_idle+0xad/0x1f0
> [37335.739945] [<ffffffff8100a743>] cpu_idle+0xb3/0x100
> [37335.739948] [<ffffffff8167d7d2>] start_secondary+0x1d7/0x1de
> [37515.727179] INFO: rcu_sched self-detected stall on CPU { 24}
> (t=1005087 jiffies g=1985385 c=1985384 q=20855557)
> [37515.727246] Pid: 0, comm: swapper/24 Tainted: G W 3.8.2 #2
> [37515.727249] Call Trace:
> [37515.727251] <IRQ> [<ffffffff810bea1e>]
> rcu_check_callbacks+0x21e/0x7c0
> [37515.727265] [<ffffffff8107f518>] ? account_system_time+0xe8/0x1e0
> [37515.727271] [<ffffffff81058f48>] update_process_times+0x48/0x90
> [37515.727275] [<ffffffff81095e06>] tick_sched_timer+0x56/0x130
> [37515.727279] [<ffffffff8107099d>] __run_hrtimer+0x7d/0x1c0
> [37515.727281] [<ffffffff81095db0>] ? tick_setup_sched_timer+0x110/0x110
> [37515.727283] [<ffffffff81070d56>] hrtimer_interrupt+0xf6/0x230
> [37515.727289] [<ffffffff81690429>] smp_apic_timer_interrupt+0x69/0x99
> [37515.727292] [<ffffffff8168f80a>] apic_timer_interrupt+0x6a/0x70
> [37515.727296] [<ffffffff815d3deb>] ?
> __inet_lookup_established+0xcb/0x2d0
> [37515.727298] [<ffffffff815cab80>] ? inet_del_protocol+0x40/0x40
> [37515.727302] [<ffffffff815f078c>] tcp_v4_early_demux+0xac/0x170
> [37515.727304] [<ffffffff815caccd>] ip_rcv_finish+0x14d/0x360
> [37515.727306] [<ffffffff815cb246>] ip_rcv+0x226/0x310
> [37515.727310] [<ffffffff815841a2>] __netif_receive_skb+0x492/0x640
> [37515.727312] [<ffffffff8158455d>] netif_receive_skb+0x2d/0x90
> [37515.727315] [<ffffffff815ed450>] ? tcp4_gro_receive+0xb0/0x130
> [37515.727317] [<ffffffff81584655>] napi_gro_complete+0x95/0xe0
> [37515.727319] [<ffffffff81584956>] dev_gro_receive+0x2b6/0x3b0
> [37515.727322] [<ffffffff8158508b>] napi_gro_receive+0x5b/0x130
> [37515.727330] [<ffffffffa01db04a>] ixgbe_poll+0x54a/0x1180 [ixgbe]
> [37515.727334] [<ffffffff810792fa>] ? enqueue_task+0x6a/0x80
> [37515.727336] [<ffffffff81584c15>] net_rx_action+0xf5/0x260
> [37515.727338] [<ffffffff810518a7>] __do_softirq+0xc7/0x230
> [37515.727341] [<ffffffff8168fe0c>] call_softirq+0x1c/0x30
> [37515.727345] [<ffffffff81004415>] do_softirq+0x55/0x90
> [37515.727346] [<ffffffff810516a5>] irq_exit+0x85/0xa0
> [37515.727349] [<ffffffff81690346>] do_IRQ+0x66/0xe0
> [37515.727354] [<ffffffff81686daa>] common_interrupt+0x6a/0x6a
> [37515.727355] <EOI> [<ffffffff8168567c>] ? __schedule+0x3ac/0x750
> [37515.727360] [<ffffffff8100b1fd>] ? mwait_idle+0xad/0x1f0
> [37515.727362] [<ffffffff8100a743>] cpu_idle+0xb3/0x100
> [37515.727365] [<ffffffff8167d7d2>] start_secondary+0x1d7/0x1de
>
> ... then swapped just does this until someone reboots the box.
>
> Apologies for the ugly paste.
>
> Thanks,
> -Dormando
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>



\
 
 \ /
  Last update: 2013-03-05 05:21    [W:0.084 / U:0.328 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site