Messages in this thread | | | From | Heiner Kallweit <> | Subject | Recent kernel change causing "rcu_sched detected stalls on CPUs/tasks" | Date | Thu, 22 Mar 2018 20:43:39 +0100 |
| |
Recently I started to get the following sporadic errors. Maybe related: When working with putty on the console then console frequently hangs for few seconds. linux-next from March 2nd still works fine. Platform is x86_64, Zotac CI-321 with Intel 2961Y.
[ 9755.298719] INFO: rcu_sched detected stalls on CPUs/tasks: [ 9755.298823] 0-...!: (9 GPs behind) idle=018/0/0 softirq=38150/38158 fqs=0 [ 9755.299006] (detected by 1, t=39181 jiffies, g=16091, c=16090, q=10) [ 9755.299068] Sending NMI from CPU 1 to CPUs 0: [ 9755.299126] NMI backtrace for cpu 0 skipped: idling at intel_idle+0x76/0x120 [ 9755.300139] rcu_sched kthread starved for 39182 jiffies! g16091 c16090 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1 [ 9755.300215] RCU grace-period kthread stack dump: [ 9755.300254] rcu_sched I 0 10 2 0x80000000 [ 9755.300307] Call Trace: [ 9755.300345] __schedule+0x27e/0xac0 [ 9755.300387] ? trace_hardirqs_on_caller+0xef/0x180 [ 9755.300432] schedule+0x2e/0x90 [ 9755.300465] schedule_timeout+0x1d4/0x570 [ 9755.300510] ? __next_timer_interrupt+0xd0/0xd0 [ 9755.300553] rcu_gp_kthread+0x491/0x7d0 [ 9755.300594] kthread+0x127/0x140 [ 9755.300628] ? call_rcu_sched+0x20/0x20 [ 9755.300667] ? kthread_create_worker_on_cpu+0x40/0x40 [ 9755.300713] ret_from_fork+0x3a/0x50 [10478.726348] INFO: rcu_sched self-detected stall on CPU [10478.726384] INFO: rcu_sched detected stalls on CPUs/tasks: [10478.726417] 0-...!: (1 GPs behind) idle=d36/0/1 softirq=38536/38591 fqs=0 [10478.726511] [10478.726523] 0-...!: (1 GPs behind) idle=d36/0/1 softirq=38536/38591 fqs=0 [10478.726591] (t=42485 jiffies g=16264 c=16263 q=6) [10478.726605] rcu_sched kthread starved for 42485 jiffies! g16264 c16263 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1 [10478.726641] [10478.726712] RCU grace-period kthread stack dump: [10478.726718] rcu_sched I [10478.726738] (detected by 1, t=42485 jiffies, g=16264, c=16263, q=6) [10478.726746] Sending NMI from CPU 1 to CPUs 0: [10478.726783] 0 10 2 0x80000000 [10478.727240] Call Trace: [10478.727273] __schedule+0x27e/0xac0 [10478.727311] ? trace_hardirqs_on_caller+0xef/0x180 [10478.727357] schedule+0x2e/0x90 [10478.727390] schedule_timeout+0x1d4/0x570 [10478.727433] ? __next_timer_interrupt+0xd0/0xd0 [10478.727475] rcu_gp_kthread+0x491/0x7d0 [10478.727517] kthread+0x127/0x140 [10478.727551] ? call_rcu_sched+0x20/0x20 [10478.727590] ? kthread_create_worker_on_cpu+0x40/0x40 [10478.727635] ret_from_fork+0x3a/0x50 [10478.727829] NMI backtrace for cpu 0 [10478.727833] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.16.0-rc6-next-20180321+ #2 [10478.727835] Hardware name: ZOTAC ZBOX-CI321NANO/ZBOX-CI321NANO, BIOS B246P105 06/01/2015 [10478.727837] RIP: 0010:console_unlock+0x79/0x640 [10478.727839] RSP: 0018:ffff96349ba03d00 EFLAGS: 00000097 [10478.727844] RAX: ffffffff8c2b9a1b RBX: 0000000000000046 RCX: 000000001da029fb [10478.727846] RDX: 0000000000000001 RSI: 00000000683da59d RDI: ffffffff8d04e1e0 [10478.727848] RBP: ffff96349ba03d38 R08: 0000000000000294 R09: 0000000000000000 [10478.727850] R10: ffff96349ba03c70 R11: 000000000000000c R12: 0000000000000000 [10478.727853] R13: 0000000000000046 R14: ffffffff8d0d9fe0 R15: 000000000000001d [10478.727855] FS: 0000000000000000(0000) GS:ffff96349ba00000(0000) knlGS:0000000000000000 [10478.727857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10478.727859] CR2: 00007fc0e849b010 CR3: 000000008c40f004 CR4: 00000000000606f0 [10478.727861] Call Trace: [10478.727862] <IRQ> [10478.727864] vprintk_emit+0x223/0x410 [10478.727866] vprintk_default+0x1a/0x20 [10478.727868] vprintk_func+0x22/0x60 [10478.727870] printk+0x3e/0x46 [10478.727871] sched_show_task+0x155/0x220 [10478.727873] rcu_check_gp_kthread_starvation+0xb9/0xda [10478.727875] rcu_check_callbacks+0x743/0x980 [10478.727877] ? trace_hardirqs_off+0xd/0x10 [10478.727879] ? tick_sched_do_timer+0x40/0x40 [10478.727881] update_process_times+0x2a/0x50 [10478.727882] tick_sched_handle+0x34/0x40 [10478.727884] tick_sched_timer+0x34/0x80 [10478.727886] __hrtimer_run_queues+0x101/0x520 [10478.727888] ? trace_hardirqs_off+0xd/0x10 [10478.727890] hrtimer_interrupt+0xc9/0x210 [10478.727892] smp_apic_timer_interrupt+0x79/0x2b0 [10478.727893] apic_timer_interrupt+0xf/0x20 [10478.727895] </IRQ> [10478.727897] RIP: 0010:cpuidle_enter_state+0xa5/0x370 [10478.727899] RSP: 0018:ffffffff8d003e28 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10478.727903] RAX: ffffffff8d014500 RBX: 000000000000695a RCX: 0000000000000000 [10478.727905] RDX: 0000000000000046 RSI: 0000000000000001 RDI: ffffffff8d014500 [10478.727907] RBP: ffffffff8d003e68 R08: 0000000000065840 R09: 0000000000000000 [10478.727910] R10: 0000000000000000 R11: 0000000000000000 R12: ffff96349ba28770 [10478.727912] R13: 0000000000000001 R14: ffffffff8d082e38 R15: 0000000000000000 [10478.727914] ? cpuidle_enter_state+0xa0/0x370 [10478.727915] cpuidle_enter+0x12/0x20 [10478.727917] call_cpuidle+0x1e/0x40 [10478.727919] do_idle+0x179/0x1c0 [10478.727921] cpu_startup_entry+0x18/0x20 [10478.727922] rest_init+0xc2/0xd0 [10478.727924] start_kernel+0x3e8/0x3f5 [10478.727926] x86_64_start_reservations+0x24/0x26 [10478.727928] x86_64_start_kernel+0x6f/0x72 [10478.727930] secondary_startup_64+0xa5/0xb0 [10478.727931] Code: 00 00 9c 8f 45 d0 fa e8 76 1d ff ff e8 31 1a 00 00 48 c7 c7 e0 e1 04 8d e8 25 74 4f 00 4c 8b 05 56 0a b2 01 4c 39 05 f7 01 ae 01 <74> 0b 4c 89 05 ee 01 ae 01 c6 45 cf 01 48 8b 0d 2b 0a b2 01 48 [10478.728026] rcu_sched kthread starved for 42485 jiffies! g16264 c16263 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=1 [10478.728028] NMI backtrace for cpu 0 [10478.728037] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.16.0-rc6-next-20180321+ #2 [10478.728061] RCU grace-period kthread stack dump: [10478.728067] rcu_sched R [10478.728122] Hardware name: ZOTAC ZBOX-CI321NANO/ZBOX-CI321NANO, BIOS B246P105 06/01/2015 [10478.728126] Call Trace: [10478.728182] running task 0 10 2 0x80000000 [10478.728224] <IRQ> [10478.728262] Call Trace: [10478.728279] __schedule+0x27e/0xac0 [10478.728321] dump_stack+0x70/0xa5 [10478.728332] nmi_cpu_backtrace+0xbe/0xc0 [10478.728388] ? trace_hardirqs_on_caller+0xef/0x180 [10478.728433] ? lapic_can_unplug_cpu+0xa0/0xa0 [10478.728443] nmi_trigger_cpumask_backtrace+0x98/0xd0 [10478.728497] schedule+0x2e/0x90 [10478.728548] arch_trigger_cpumask_backtrace+0x14/0x20 [10478.728605] schedule_timeout+0x1d4/0x570 [10478.728617] ? __next_timer_interrupt+0xd0/0xd0 [10478.728656] rcu_dump_cpu_stacks+0xa7/0xd9 [10478.728665] rcu_check_callbacks+0x74b/0x980 [10478.728717] rcu_gp_kthread+0x491/0x7d0 [10478.728741] ? trace_hardirqs_off+0xd/0x10 [10478.728750] ? tick_sched_do_timer+0x40/0x40 [10478.728774] kthread+0x127/0x140 [10478.728800] update_process_times+0x2a/0x50 [10478.728809] tick_sched_handle+0x34/0x40 [10478.728839] ? call_rcu_sched+0x20/0x20 [10478.728848] ? kthread_create_worker_on_cpu+0x40/0x40 [10478.728876] tick_sched_timer+0x34/0x80 [10478.728883] __hrtimer_run_queues+0x101/0x520 [10478.728914] ret_from_fork+0x3a/0x50 [10478.728944] ? trace_hardirqs_off+0xd/0x10 [10478.728953] hrtimer_interrupt+0xc9/0x210 [10478.731180] smp_apic_timer_interrupt+0x79/0x2b0 [10478.731223] apic_timer_interrupt+0xf/0x20 [10478.731259] </IRQ> [10478.731287] RIP: 0010:cpuidle_enter_state+0xa5/0x370 [10478.731328] RSP: 0018:ffffffff8d003e28 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10478.731392] RAX: ffffffff8d014500 RBX: 000000000000695a RCX: 0000000000000000 [10478.731446] RDX: 0000000000000046 RSI: 0000000000000001 RDI: ffffffff8d014500 [10478.731500] RBP: ffffffff8d003e68 R08: 0000000000065840 R09: 0000000000000000 [10478.731554] R10: 0000000000000000 R11: 0000000000000000 R12: ffff96349ba28770 [10478.731607] R13: 0000000000000001 R14: ffffffff8d082e38 R15: 0000000000000000 [10478.731666] ? cpuidle_enter_state+0xa0/0x370 [10478.731706] cpuidle_enter+0x12/0x20 [10478.731742] call_cpuidle+0x1e/0x40 [10478.731776] do_idle+0x179/0x1c0 [10478.731810] cpu_startup_entry+0x18/0x20 [10478.731850] rest_init+0xc2/0xd0 [10478.731885] start_kernel+0x3e8/0x3f5 [10478.731922] x86_64_start_reservations+0x24/0x26 [10478.731965] x86_64_start_kernel+0x6f/0x72 [10478.732006] secondary_startup_64+0xa5/0xb0 [15106.895191] INFO: rcu_sched detected stalls on CPUs/tasks: [15106.895264] 0-...!: (1 GPs behind) idle=850/0/0 softirq=46299/46345 fqs=0 [15106.895318] (detected by 1, t=49858 jiffies, g=17103, c=17102, q=6) [15106.895378] Sending NMI from CPU 1 to CPUs 0: [15106.895436] NMI backtrace for cpu 0 skipped: idling at intel_idle+0x76/0x120 [15106.896428] rcu_sched kthread starved for 49859 jiffies! g17103 c17102 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1 [15106.896507] RCU grace-period kthread stack dump: [15106.896546] rcu_sched I 0 10 2 0x80000000 [15106.896599] Call Trace: [15106.896639] __schedule+0x27e/0xac0 [15106.896680] ? trace_hardirqs_on_caller+0xef/0x180 [15106.896726] schedule+0x2e/0x90 [15106.896760] schedule_timeout+0x1d4/0x570 [15106.896805] ? __next_timer_interrupt+0xd0/0xd0 [15106.896848] rcu_gp_kthread+0x491/0x7d0 [15106.896889] kthread+0x127/0x140 [15106.896923] ? call_rcu_sched+0x20/0x20 [15106.896962] ? kthread_create_worker_on_cpu+0x40/0x40 [15106.897009] ret_from_fork+0x3a/0x50 [18580.113132] INFO: rcu_sched detected stalls on CPUs/tasks: [18580.113213] 0-...!: (1 GPs behind) idle=d36/0/1 softirq=52012/52014 fqs=0 [18580.113272] (detected by 1, t=42478 jiffies, g=17532, c=17531, q=6) [18580.113342] Sending NMI from CPU 1 to CPUs 0: [18580.113397] NMI backtrace for cpu 0 [18580.113406] CPU: 0 PID: 25 Comm: kworker/0:2 Not tainted 4.16.0-rc6-next-20180321+ #2 [18580.113409] Hardware name: ZOTAC ZBOX-CI321NANO/ZBOX-CI321NANO, BIOS B246P105 06/01/2015 [18580.113426] Workqueue: events check_corruption [18580.113438] RIP: 0010:check_for_bios_corruption+0x5a/0xf0 [18580.113442] RSP: 0018:ffffae9a006fbe10 EFLAGS: 00000246 [18580.113447] RAX: 0000000000000000 RBX: ffff963380008c18 RCX: 0000000000000000 [18580.113449] RDX: 0000000000000001 RSI: ffffffff8d04f8a0 RDI: ffffffff8d02f600 [18580.113452] RBP: ffffae9a006fbe40 R08: ffffffff80000000 R09: 0000000000000000 [18580.113455] R10: ffffae9a006fbde8 R11: 0000000000000004 R12: 00000000000073e8 [18580.113458] R13: ffffffff8d202340 R14: 0000000000000000 R15: 0000000080000000 [18580.113462] FS: 0000000000000000(0000) GS:ffff96349ba00000(0000) knlGS:0000000000000000 [18580.113466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18580.113469] CR2: 00007fc0e793d3b0 CR3: 000000008c40f005 CR4: 00000000000606f0 [18580.113471] Call Trace: [18580.113484] check_corruption+0x9/0x40 [18580.113495] process_one_work+0x24c/0x670 [18580.113504] worker_thread+0x35/0x3f0 [18580.113512] kthread+0x127/0x140 [18580.113520] ? process_one_work+0x670/0x670 [18580.113527] ? kthread_create_worker_on_cpu+0x40/0x40 [18580.113536] ret_from_fork+0x3a/0x50 [18580.113539] Code: 8d 53 45 31 f6 41 bf 00 00 00 80 48 83 ec 08 4d 8b 65 08 48 8b 1d 37 c3 ba 00 49 03 5d 00 4d 85 e4 74 12 48 8b 0b 48 85 c9 75 50 <48> 83 c3 08 49 83 ec 08 75 ee 41 83 c6 01 49 83 c5 10 44 39 35 [18580.114401] rcu_sched kthread starved for 42479 jiffies! g17532 c17531 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1 [18580.115131] RCU grace-period kthread stack dump: [18580.115171] rcu_sched R running task 0 10 2 0x80000000 [18580.115239] Call Trace: [18580.115279] __schedule+0x27e/0xac0 [18580.115321] ? trace_hardirqs_on_caller+0xef/0x180 [18580.115368] schedule+0x2e/0x90 [18580.115401] schedule_timeout+0x1d4/0x570 [18580.115447] ? __next_timer_interrupt+0xd0/0xd0 [18580.115490] rcu_gp_kthread+0x491/0x7d0 [18580.115531] kthread+0x127/0x140 [18580.115565] ? call_rcu_sched+0x20/0x20 [18580.115604] ? kthread_create_worker_on_cpu+0x40/0x40 [18580.115651] ret_from_fork+0x3a/0x50 [19077.707886] INFO: rcu_sched detected stalls on CPUs/tasks: [19077.707958] 0-...!: (1 GPs behind) idle=0b4/0/0 softirq=52380/52381 fqs=0 [19077.708012] (detected by 1, t=59990 jiffies, g=17586, c=17585, q=8) [19077.708073] Sending NMI from CPU 1 to CPUs 0: [19077.708131] NMI backtrace for cpu 0 skipped: idling at intel_idle+0x76/0x120 [19077.709123] rcu_sched kthread starved for 59991 jiffies! g17586 c17585 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1 [19077.709201] RCU grace-period kthread stack dump: [19077.709241] rcu_sched I 0 10 2 0x80000000 [19077.709294] Call Trace: [19077.709334] __schedule+0x27e/0xac0 [19077.709376] ? trace_hardirqs_on_caller+0xef/0x180 [19077.709421] schedule+0x2e/0x90 [19077.709455] schedule_timeout+0x1d4/0x570 [19077.709500] ? __next_timer_interrupt+0xd0/0xd0 [19077.709543] rcu_gp_kthread+0x491/0x7d0 [19077.709584] kthread+0x127/0x140 [19077.709618] ? call_rcu_sched+0x20/0x20 [19077.709657] ? kthread_create_worker_on_cpu+0x40/0x40 [19077.709704] ret_from_fork+0x3a/0x50 [22729.821902] INFO: rcu_sched self-detected stall on CPU [22729.821937] INFO: rcu_sched detected stalls on CPUs/tasks: [22729.821971] 0-...!: (1 GPs behind) idle=24a/1/4611686018427387906 softirq=56015/56021 fqs=0 [22729.822016] (t=51485 jiffies g=18062 c=18061 q=5) [22729.822099] 0-...!: (1 GPs behind) idle=24a/1/4611686018427387906 softirq=56015/56021 fqs=0 [22729.822132] rcu_sched kthread starved for 51485 jiffies! g18062 c18061 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1 [22729.822137] RCU grace-period kthread stack dump: [22729.822200] [22729.822273] rcu_sched I 0 10 2 0x80000000 [22729.822316] (detected by 1, t=51485 jiffies, g=18062, c=18061, q=5) [22729.822324] Sending NMI from CPU 1 to CPUs 0: [22729.822341] Call Trace: [22729.822832] __schedule+0x27e/0xac0 [22729.822870] ? trace_hardirqs_on_caller+0xef/0x180 [22729.822916] schedule+0x2e/0x90 [22729.822949] schedule_timeout+0x1d4/0x570 [22729.822991] ? __next_timer_interrupt+0xd0/0xd0 [22729.823033] rcu_gp_kthread+0x491/0x7d0 [22729.823073] kthread+0x127/0x140 [22729.823107] ? call_rcu_sched+0x20/0x20 [22729.823145] ? kthread_create_worker_on_cpu+0x40/0x40 [22729.823191] ret_from_fork+0x3a/0x50 [22729.823385] NMI backtrace for cpu 0 [22729.823388] CPU: 0 PID: 25 Comm: kworker/0:2 Not tainted 4.16.0-rc6-next-20180321+ #2 [22729.823391] Hardware name: ZOTAC ZBOX-CI321NANO/ZBOX-CI321NANO, BIOS B246P105 06/01/2015 [22729.823393] RIP: 0010:console_unlock+0xd1/0x640 [22729.823395] RSP: 0018:ffff96349ba03c20 EFLAGS: 00000082 [22729.823399] RAX: ffffffff8c2b9a1b RBX: ffffffff8ddaa62c RCX: 0000000000000361 [22729.823402] RDX: ffffffff8dd9a450 RSI: 0000000000000048 RDI: 00000000000101dc [22729.823404] RBP: ffff96349ba03c58 R08: 0000000000000364 R09: 0000000000000007 [22729.823406] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [22729.823408] R13: 0000000000000046 R14: ffffffff8d0d9fe0 R15: 000000000000000b [22729.823410] FS: 0000000000000000(0000) GS:ffff96349ba00000(0000) knlGS:0000000000000000 [22729.823413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22729.823415] CR2: 0000000000000130 CR3: 000000008c40f006 CR4: 00000000000606f0 [22729.823416] Call Trace: [22729.823418] <IRQ> [22729.823420] vprintk_emit+0x223/0x410 [22729.823422] vprintk_default+0x1a/0x20 [22729.823424] vprintk_func+0x22/0x60 [22729.823425] printk+0x3e/0x46 [22729.823427] ? console_unlock+0x5fd/0x640 [22729.823429] show_trace_log_lvl+0x6e/0x300 [22729.823431] ? vprintk_default+0x1a/0x20 [22729.823432] ? vprintk_func+0x22/0x60 [22729.823434] show_stack+0x2f/0x40 [22729.823436] sched_show_task+0x16e/0x220 [22729.823438] rcu_check_gp_kthread_starvation+0xb9/0xda [22729.823440] rcu_check_callbacks+0x743/0x980 [22729.823442] ? trace_hardirqs_off+0xd/0x10 [22729.823443] ? tick_sched_do_timer+0x40/0x40 [22729.823445] update_process_times+0x2a/0x50 [22729.823447] tick_sched_handle+0x34/0x40 [22729.823449] tick_sched_timer+0x34/0x80 [22729.823451] __hrtimer_run_queues+0x101/0x520 [22729.823453] ? trace_hardirqs_off+0xd/0x10 [22729.823454] hrtimer_interrupt+0xc9/0x210 [22729.823456] smp_apic_timer_interrupt+0x79/0x2b0 [22729.823458] apic_timer_interrupt+0xf/0x20 [22729.823460] </IRQ> [22729.823462] RIP: 0010:_raw_spin_unlock_irq+0x2f/0x40 [22729.823464] RSP: 0018:ffffae9a006fbeb0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13 [22729.823468] RAX: ffff96349afa2b00 RBX: ffff96349ba20700 RCX: 0000000000000000 [22729.823470] RDX: 0000000000000046 RSI: 0000000000000001 RDI: ffff96349afa2b00 [22729.823472] RBP: ffffae9a006fbeb8 R08: 0000000000000000 R09: 0000000000000000 [22729.823474] R10: 0000000000000000 R11: 0000000000000000 R12: ffff96349af25c30 [22729.823477] R13: ffff96349ba20700 R14: ffff96349ba20750 R15: ffff96349af25c00 [22729.823478] worker_thread+0xba/0x3f0 [22729.823480] kthread+0x127/0x140 [22729.823482] ? process_one_work+0x670/0x670 [22729.823484] ? kthread_create_worker_on_cpu+0x40/0x40 [22729.823486] ret_from_fork+0x3a/0x50 [22729.823487] Code: 03 00 00 44 0f b6 15 23 06 e2 00 8b 3d f9 09 b2 01 45 31 db 48 8b 15 f7 46 d9 00 44 8b 0d 38 48 d9 00 89 fb 48 01 d3 0f b7 73 08 <66> 85 f6 48 0f 44 da 0f b6 43 0f c0 e8 05 0f b6 c0 44 39 c8 7c [22729.823582] rcu_sched kthread starved for 51485 jiffies! g18062 c18061 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=1 [22729.823584] NMI backtrace for cpu 0 [22729.823593] CPU: 0 PID: 25 Comm: kworker/0:2 Not tainted 4.16.0-rc6-next-20180321+ #2 [22729.823617] RCU grace-period kthread stack dump: [22729.823623] rcu_sched R [22729.823679] Hardware name: ZOTAC ZBOX-CI321NANO/ZBOX-CI321NANO, BIOS B246P105 06/01/2015 [22729.823695] Call Trace: [22729.823739] running task 0 10 2 0x80000000 [22729.823782] <IRQ> [22729.823819] Call Trace: [22729.823836] __schedule+0x27e/0xac0 [22729.823878] dump_stack+0x70/0xa5 [22729.823889] nmi_cpu_backtrace+0xbe/0xc0 [22729.823944] ? trace_hardirqs_on_caller+0xef/0x180 [22729.823990] ? lapic_can_unplug_cpu+0xa0/0xa0 [22729.824000] nmi_trigger_cpumask_backtrace+0x98/0xd0 [22729.824054] schedule+0x2e/0x90 [22729.824104] arch_trigger_cpumask_backtrace+0x14/0x20 [22729.824161] schedule_timeout+0x1d4/0x570 [22729.824173] ? __next_timer_interrupt+0xd0/0xd0 [22729.824212] rcu_dump_cpu_stacks+0xa7/0xd9 [22729.824221] rcu_check_callbacks+0x74b/0x980 [22729.824273] rcu_gp_kthread+0x491/0x7d0 [22729.824296] ? trace_hardirqs_off+0xd/0x10 [22729.824306] ? tick_sched_do_timer+0x40/0x40 [22729.824330] kthread+0x127/0x140 [22729.824356] update_process_times+0x2a/0x50 [22729.824365] tick_sched_handle+0x34/0x40 [22729.824394] ? call_rcu_sched+0x20/0x20 [22729.824404] ? kthread_create_worker_on_cpu+0x40/0x40 [22729.824431] tick_sched_timer+0x34/0x80 [22729.824439] __hrtimer_run_queues+0x101/0x520 [22729.824469] ret_from_fork+0x3a/0x50 [22729.824503] ? trace_hardirqs_off+0xd/0x10 [22729.826703] hrtimer_interrupt+0xc9/0x210 [22729.826744] smp_apic_timer_interrupt+0x79/0x2b0 [22729.826787] apic_timer_interrupt+0xf/0x20 [22729.826822] </IRQ> [22729.826850] RIP: 0010:_raw_spin_unlock_irq+0x2f/0x40 [22729.826891] RSP: 0018:ffffae9a006fbeb0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13 [22729.826955] RAX: ffff96349afa2b00 RBX: ffff96349ba20700 RCX: 0000000000000000 [22729.827009] RDX: 0000000000000046 RSI: 0000000000000001 RDI: ffff96349afa2b00 [22729.827063] RBP: ffffae9a006fbeb8 R08: 0000000000000000 R09: 0000000000000000 [22729.827117] R10: 0000000000000000 R11: 0000000000000000 R12: ffff96349af25c30 [22729.827171] R13: ffff96349ba20700 R14: ffff96349ba20750 R15: ffff96349af25c00 [22729.827232] worker_thread+0xba/0x3f0 [22729.827270] kthread+0x127/0x140 [22729.827306] ? process_one_work+0x670/0x670 [22729.827346] ? kthread_create_worker_on_cpu+0x40/0x40 [22729.827391] ret_from_fork+0x3a/0x50 [45584.732020] clocksource: timekeeping watchdog on CPU0: Marking clocksource 'tsc' as unstable because the skew is too large: [45584.732178] clocksource: 'hpet' wd_now: f7640fac wd_last: d616ddff mask: ffffffff [45584.732234] clocksource: 'tsc' cs_now: 2d80e82d6501 cs_last: 2d2a49cd275b mask: ffffffffffffffff [45584.732585] tsc: Marking TSC unstable due to clocksource watchdog [45584.733147] TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'. [45584.733275] sched_clock: Marking unstable (45576934247406, 7737020246)<-(45584732356690, 785036) [45584.672583] clocksource: Switched to clocksource hpet [70247.757041] INFO: rcu_sched detected stalls on CPUs/tasks: [70247.757114] 0-...!: (9 GPs behind) idle=2f4/0/0 softirq=127686/127689 fqs=0 [70247.757169] (detected by 1, t=45569 jiffies, g=37994, c=37993, q=7) [70247.757230] Sending NMI from CPU 1 to CPUs 0: [70247.757288] NMI backtrace for cpu 0 skipped: idling at intel_idle+0x76/0x120 [70247.757908] rcu_sched kthread starved for 45570 jiffies! g37994 c37993 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1 [70247.757908] RCU grace-period kthread stack dump: [70247.757908] rcu_sched I 0 10 2 0x80000000 [70247.757908] Call Trace: [70247.757908] __schedule+0x27e/0xac0 [70247.757908] ? trace_hardirqs_on_caller+0xef/0x180 [70247.757908] schedule+0x2e/0x90 [70247.757908] schedule_timeout+0x1d4/0x570 [70247.757908] ? __next_timer_interrupt+0xd0/0xd0 [70247.757908] rcu_gp_kthread+0x491/0x7d0 [70247.757908] kthread+0x127/0x140 [70247.757908] ? call_rcu_sched+0x20/0x20 [70247.757908] ? kthread_create_worker_on_cpu+0x40/0x40 [70247.757908] ret_from_fork+0x3a/0x50
| |