lkml.org 
[lkml]   [2015]   [Sep]   [1]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectPossible deadlock related to CPU hotplug and kernfs
Date
Hi Rafael and Tejun,
When running CPU hotplug tests, it triggers an lockdep warning
as follow. The two possible deadlock paths are:
1) echo x > /sys/devices/system/cpu/cpux/online
->kernfs_fop_write()
->kernfs_get_active()
1.a) ->rwsem_acquire_read(&kn->dep_map, 0, 1, _RET_IP_);
->cpu_up()
1.b) ->cpu_hotplug_begin()[lock_map_acquire(&cpu_hotplug.dep_map)]
2) hardware triggers hotplug evetns
->acpi_device_hotplug()
->acpi_processor_remove()
2.a) ->cpu_hotplug_begin()[lock_map_acquire(&cpu_hotplug.dep_map)]
->unregister_cpu()
->device_del()
->kernfs_remove_by_name_ns()
->__kernfs_remove()
->kernfs_drain()
2.b) ->rwsem_acquire(&kn->dep_map, 0, 0, _RET_IP_)

So there is a possible deadlock scenario among 1.a, 1.b, 2.a and 2.b.
I'm not familiar with kernfs, so could you please help to comment:
1) whether is a real deadlock issue?
2) any recommended way to get it fixed?
Thanks!
Gerry

Full lockdep warnings:
[ 310.309391] [ INFO: possible circular locking dependency detected ]
[ 310.316462] 4.2.0-rc8+ #7 Not tainted
[ 310.320613] -------------------------------------------------------
[ 310.327684] kworker/u288:3/388 is trying to acquire lock:
[ 310.333780] (s_active#97){++++.+}, at: [<ffffffff812bd989>]
kernfs_remove_by_name_ns+0x49/0xb0
[ 310.343885]
[ 310.343885] but task is already holding lock:
[ 310.350466] (cpu_hotplug.lock#2){+.+.+.}, at: [<ffffffff81080aab>]
cpu_hotplug_begin+0x7b/0xc0
[ 310.360564]
[ 310.360564] which lock already depends on the new lock.
[ 310.360564]
[ 310.369766]
[ 310.369766] the existing dependency chain (in reverse order) is:
[ 310.378198]
[ 310.378198] -> #3 (cpu_hotplug.lock#2){+.+.+.}:
[ 310.383821] [<ffffffff810df04d>] lock_acquire+0xdd/0x2a0
[ 310.390591] [<ffffffff818644a0>] mutex_lock_nested+0x70/0x3e0
[ 310.397847] [<ffffffff81080aab>] cpu_hotplug_begin+0x7b/0xc0
[ 310.405004] [<ffffffff81080b61>] _cpu_up+0x31/0x140
[ 310.411285] [<ffffffff81080cec>] cpu_up+0x7c/0xa0
[ 310.417362] [<ffffffff821859cb>] smp_init+0x86/0x88
[ 310.423647] [<ffffffff82160181>] kernel_init_freeable+0x171/0x286
[ 310.431292] [<ffffffff8185228e>] kernel_init+0xe/0xe0
[ 310.437771] [<ffffffff81869e5f>] ret_from_fork+0x3f/0x70
[ 310.444540]
[ 310.444540] -> #2 (cpu_hotplug.lock){++++++}:
[ 310.449957] [<ffffffff810df04d>] lock_acquire+0xdd/0x2a0
[ 310.456714] [<ffffffff81080a9d>] cpu_hotplug_begin+0x6d/0xc0
[ 310.463871] [<ffffffff81080b61>] _cpu_up+0x31/0x140
[ 310.470143] [<ffffffff81080cec>] cpu_up+0x7c/0xa0
[ 310.476228] [<ffffffff821859cb>] smp_init+0x86/0x88
[ 310.482509] [<ffffffff82160181>] kernel_init_freeable+0x171/0x286
[ 310.490153] [<ffffffff8185228e>] kernel_init+0xe/0xe0
[ 310.496628] [<ffffffff81869e5f>] ret_from_fork+0x3f/0x70
[ 310.503393]
[ 310.503393] -> #1 (cpu_add_remove_lock){+.+.+.}:
[ 310.509099] [<ffffffff810df04d>] lock_acquire+0xdd/0x2a0
[ 310.515866] [<ffffffff811e1134>] __might_fault+0x84/0xb0
[ 310.522635] [<ffffffff812beb6f>] kernfs_fop_write+0x8f/0x190
[ 310.529793] [<ffffffff81233b68>] __vfs_write+0x28/0xe0
[ 310.536368] [<ffffffff812342ac>] vfs_write+0xac/0x1a0
[ 310.542833] [<ffffffff81235049>] SyS_write+0x49/0xb0
[ 310.549212] [<ffffffff818699f2>]
entry_SYSCALL_64_fastpath+0x16/0x7a
[ 310.557149]
[ 310.557149] -> #0 (s_active#97){++++.+}:
[ 310.562135] [<ffffffff810de269>] __lock_acquire+0x21b9/0x21c0
[ 310.569391] [<ffffffff810df04d>] lock_acquire+0xdd/0x2a0
[ 310.576159] [<ffffffff812bc7a1>] __kernfs_remove+0x231/0x330
[ 310.583318] [<ffffffff812bd989>]
kernfs_remove_by_name_ns+0x49/0xb0
[ 310.591154] [<ffffffff812bf3c5>] sysfs_remove_file_ns+0x15/0x20
[ 310.598594] [<ffffffff8157490e>] device_remove_attrs+0x3e/0x80
[ 310.605948] [<ffffffff815752a8>] device_del+0x138/0x270
[ 310.612617] [<ffffffff81575402>] device_unregister+0x22/0x70
[ 310.619767] [<ffffffff8157cfa9>] unregister_cpu+0x39/0x60
[ 310.626622] [<ffffffff81023e73>] arch_unregister_cpu+0x23/0x30
[ 310.633974] [<ffffffff814bab67>] acpi_processor_remove+0x91/0xca
[ 310.641524] [<ffffffff814b82e3>] acpi_bus_trim+0x5a/0x8d
[ 310.648292] [<ffffffff814b82c1>] acpi_bus_trim+0x38/0x8d
[ 310.655060] [<ffffffff814b8333>]
acpi_scan_device_not_present+0x1d/0x3d
[ 310.663312] [<ffffffff814b9e05>] acpi_scan_bus_check+0x29/0xa2
[ 310.670654] [<ffffffff814b9f17>] acpi_device_hotplug+0x99/0x3fa
[ 310.678103] [<ffffffff814b33ba>] acpi_hotplug_work_fn+0x1f/0x2b
[ 310.685555] [<ffffffff810a0241>] process_one_work+0x1f1/0x7c0
[ 310.692814] [<ffffffff810a0879>] worker_thread+0x69/0x480
[ 310.699677] [<ffffffff810a71af>] kthread+0x11f/0x140
[ 310.706046] [<ffffffff81869e5f>] ret_from_fork+0x3f/0x70
[ 310.712815]
[ 310.712815] other info that might help us debug this:
[ 310.712815]
[ 310.721907] Chain exists of:
[ 310.721907] s_active#97 --> cpu_hotplug.lock --> cpu_hotplug.lock#2
[ 310.721907]
[ 310.731680] Possible unsafe locking scenario:
[ 310.731680]
[ 310.738413] CPU0 CPU1
[ 310.743562] ---- ----
[ 310.748710] lock(cpu_hotplug.lock#2);
[ 310.753261] lock(cpu_hotplug.lock);
[ 310.760382] lock(cpu_hotplug.lock#2);
[ 310.767755] lock(s_active#97);
[ 310.771625]
[ 310.771625] *** DEADLOCK ***
[ 310.771625]
[ 310.778382] 7 locks held by kworker/u288:3/388:
[ 310.783530] #0: ("kacpi_hotplug"){.+.+.+}, at: [<ffffffff810a01b6>]
process_one_work+0x166/0x7c0
[ 310.793975] #1: ((&hpw->work)){+.+.+.}, at: [<ffffffff810a01b6>]
process_one_work+0x166/0x7c0
[ 310.804126] #2: (device_hotplug_lock){+.+.+.}, at:
[<ffffffff81575cc7>] lock_device_hotplug+0x17/0x20
[ 310.815057] #3: (acpi_scan_lock){+.+.+.}, at: [<ffffffff814b9eb4>]
acpi_device_hotplug+0x36/0x3fa
[ 310.825599] #4: (cpu_add_remove_lock){+.+.+.}, at:
[<ffffffff810807d7>] cpu_maps_update_begin+0x17/0x20
[ 310.836727] #5: (cpu_hotplug.lock){++++++}, at:
[<ffffffff81080a35>] cpu_hotplug_begin+0x5/0xc0
[ 310.847073] #6: (cpu_hotplug.lock#2){+.+.+.}, at:
[<ffffffff81080aab>] cpu_hotplug_begin+0x7b/0xc0
[ 310.857774]
[ 310.857774] stack backtrace:
[ 310.862754] CPU: 11 PID: 388 Comm: kworker/u288:3 Not tainted
4.2.0-rc8+ #7
[ 310.870628] Hardware name: Intel Corporation BRICKLAND/BRICKLAND,
BIOS BRHSXIN1.86B.0060.R02.1508171754 08/17/2015
[ 310.882326] Workqueue: kacpi_hotplug acpi_hotplug_work_fn
[ 310.888499] ffffffff82a39b50 ffff88042b9a38d8 ffffffff8185f0b8
0000000000000011
[ 310.897130] ffffffff82afcab0 ffff88042b9a3928 ffffffff8185c183
0000000000000007
[ 310.905762] ffff88042b9a3998 ffff88042b9a3928 ffff88042b99ab08
ffff88042b99a980
[ 310.914393] Call Trace:
[ 310.917206] [<ffffffff8185f0b8>] dump_stack+0x4c/0x65
[ 310.923039] [<ffffffff8185c183>] print_circular_bug+0x20b/0x21c
[ 310.929843] [<ffffffff810de269>] __lock_acquire+0x21b9/0x21c0
[ 310.936455] [<ffffffff810260d8>] ? native_sched_clock+0x28/0x90
[ 310.943258] [<ffffffff810df04d>] lock_acquire+0xdd/0x2a0
[ 310.949382] [<ffffffff812bd989>] ? kernfs_remove_by_name_ns+0x49/0xb0
[ 310.956769] [<ffffffff812bc7a1>] __kernfs_remove+0x231/0x330
[ 310.963280] [<ffffffff812bd989>] ? kernfs_remove_by_name_ns+0x49/0xb0
[ 310.970669] [<ffffffff812bbd67>] ? kernfs_name_hash+0x17/0xa0
[ 310.977278] [<ffffffff812bcb81>] ? kernfs_find_ns+0x81/0x140
[ 310.983792] [<ffffffff812bd989>] kernfs_remove_by_name_ns+0x49/0xb0
[ 310.990986] [<ffffffff812bf3c5>] sysfs_remove_file_ns+0x15/0x20
[ 310.997791] [<ffffffff8157490e>] device_remove_attrs+0x3e/0x80
[ 311.004498] [<ffffffff815752a8>] device_del+0x138/0x270
[ 311.010524] [<ffffffff812bd995>] ? kernfs_remove_by_name_ns+0x55/0xb0
[ 311.017914] [<ffffffff81575402>] device_unregister+0x22/0x70
[ 311.024427] [<ffffffff8157cfa9>] unregister_cpu+0x39/0x60
[ 311.030646] [<ffffffff81023e73>] arch_unregister_cpu+0x23/0x30
[ 311.037354] [<ffffffff814bab67>] acpi_processor_remove+0x91/0xca
[ 311.044257] [<ffffffff814b82e3>] acpi_bus_trim+0x5a/0x8d
[ 311.050379] [<ffffffff814b82c1>] acpi_bus_trim+0x38/0x8d
[ 311.056501] [<ffffffff814b8333>] acpi_scan_device_not_present+0x1d/0x3d
[ 311.064085] [<ffffffff814b9e05>] acpi_scan_bus_check+0x29/0xa2
[ 311.070791] [<ffffffff814b9f17>] acpi_device_hotplug+0x99/0x3fa
[ 311.077596] [<ffffffff814b33ba>] acpi_hotplug_work_fn+0x1f/0x2b
[ 311.084402] [<ffffffff810a0241>] process_one_work+0x1f1/0x7c0
[ 311.091012] [<ffffffff810a01b6>] ? process_one_work+0x166/0x7c0
[ 311.097815] [<ffffffff810a0909>] ? worker_thread+0xf9/0x480
[ 311.104231] [<ffffffff810a0879>] worker_thread+0x69/0x480
[ 311.110451] [<ffffffff810a0810>] ? process_one_work+0x7c0/0x7c0
[ 311.117256] [<ffffffff810a71af>] kthread+0x11f/0x140
[ 311.122990] [<ffffffff810a7090>] ? kthread_create_on_node+0x260/0x260
[ 311.130379] [<ffffffff81869e5f>] ret_from_fork+0x3f/0x70
[ 311.136502] [<ffffffff810a7090>] ? kthread_create_on_node+0x260/0x260


\
 
 \ /
  Last update: 2015-09-01 09:21    [W:0.077 / U:0.448 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site