Messages in this thread | | | From | Martin Hundebøll <> | Subject | [BUG] tty: n_gsm: possible circular locking dependency detected | Date | Thu, 19 Sep 2019 15:27:42 +0200 |
| |
Hi,
We've come across the below lockdep report when using n_gsm multiplexing on an imx uart.
The inverted lock scenario caused by: 1) the uart tx-ready interrupt causing the tty layer to call down into n_gsm, and 2) n_gsm transmitting data, causing it to call up into the tty layer
We have about 2-3k devices running, and get the report from ~100 devices during a 1 hour span. But we haven't been able to reproduce locally.
The devices run 4.19.22 with my recent n_gsm patches backported.
Any hints on how to dig further are appreciated.
// Martin
[ 201.633281] ====================================================== [ 201.639473] WARNING: possible circular locking dependency detected [ 201.645667] 4.19.22 #1 Not tainted [ 201.649078] ------------------------------------------------------ [ 201.655270] kworker/u2:0/7 is trying to acquire lock: [ 201.660337] a66ff7b8 (&(&gsm->tx_lock)->rlock){-.-.}, at: gsmld_write_wakeup+0x28/0x5c [ 201.668309] [ 201.668309] but task is already holding lock: [ 201.674155] 456c79ff (&port_lock_key){-.-.}, at: imx_uart_txint+0x18/0x198 [ 201.681070] [ 201.681070] which lock already depends on the new lock. [ 201.681070] [ 201.689261] [ 201.689261] the existing dependency chain (in reverse order) is: [ 201.696754] [ 201.696754] -> #1 (&port_lock_key){-.-.}: [ 201.702282] _raw_spin_lock_irqsave+0x40/0x54 [ 201.707178] uart_write_room+0x60/0xf8 [ 201.711467] tty_write_room+0x20/0x2c [ 201.715665] gsmld_output+0x20/0x94 [ 201.719690] gsm_data_kick+0x80/0x164 [ 201.723888] __gsm_data_queue+0x188/0x1a4 [ 201.728433] gsm_control_transmit+0x78/0x8c [ 201.733151] gsm_control_send+0x130/0x170 [ 201.737693] gsmld_ioctl+0x238/0x4f8 [ 201.741805] tty_ioctl+0x3a0/0xc44 [ 201.745744] do_vfs_ioctl+0xa8/0xa3c [ 201.749855] ksys_ioctl+0x3c/0x68 [ 201.753703] sys_ioctl+0x10/0x14 [ 201.757468] ret_fast_syscall+0x0/0x28 [ 201.761752] 0xbecd6ab4 [ 201.764728] [ 201.764728] -> #0 (&(&gsm->tx_lock)->rlock){-.-.}: [ 201.771036] lock_acquire+0xd0/0x1f4 [ 201.775148] _raw_spin_lock_irqsave+0x40/0x54 [ 201.780042] gsmld_write_wakeup+0x28/0x5c [ 201.784587] tty_wakeup+0x58/0x64 [ 201.788436] tty_port_default_wakeup+0x1c/0x28 [ 201.793412] tty_port_tty_wakeup+0x18/0x1c [ 201.798043] uart_write_wakeup+0x1c/0x24 [ 201.802498] imx_uart_txint+0x13c/0x198 [ 201.806868] imx_uart_int+0x100/0x1a4 [ 201.811066] __handle_irq_event_percpu+0x44/0x364 [ 201.816302] handle_irq_event_percpu+0x30/0x88 [ 201.821279] handle_irq_event+0x40/0x64 [ 201.825652] handle_fasteoi_irq+0xc4/0x174 [ 201.830282] generic_handle_irq+0x28/0x3c [ 201.834826] __handle_domain_irq+0x6c/0xe8 [ 201.839459] gic_handle_irq+0x60/0xc4 [ 201.843657] __irq_svc+0x70/0x98 [ 201.847421] _raw_spin_unlock_irq+0x30/0x34 [ 201.852141] process_one_work+0x210/0x718 [ 201.856686] worker_thread+0x2c/0x54c [ 201.860881] kthread+0x14c/0x164 [ 201.864643] ret_from_fork+0x14/0x20 [ 201.868749] (null) [ 201.871551] [ 201.871551] other info that might help us debug this: [ 201.871551] [ 201.879568] Possible unsafe locking scenario: [ 201.879568] [ 201.885499] CPU0 CPU1 [ 201.890038] ---- ---- [ 201.894574] lock(&port_lock_key); [ 201.898082] lock(&(&gsm->tx_lock)->rlock); [ 201.904890] lock(&port_lock_key); [ 201.910913] lock(&(&gsm->tx_lock)->rlock); [ 201.915200] [ 201.915200] *** DEADLOCK *** [ 201.915200] [ 201.921134] 2 locks held by kworker/u2:0/7: [ 201.925326] #0: 456c79ff (&port_lock_key){-.-.}, at: imx_uart_txint+0x18/0x198 [ 201.932683] #1: e4c7457f (&tty->ldisc_sem){++++}, at: tty_ldisc_ref+0x1c/0x44 [ 201.939948] [ 201.939948] stack backtrace: [ 201.944326] CPU: 0 PID: 7 Comm: kworker/u2:0 Not tainted 4.19.22 #1 [ 201.950604] Hardware name: Freescale i.MX6 Ultralite (Device Tree) [ 201.956800] Workqueue: events_unbound flush_to_ldisc [ 201.961782] Backtrace: [ 201.964255] [<c010ebd4>] (dump_backtrace) from [<c010ef5c>] (show_stack+0x18/0x1c) [ 201.971840] r7:00000000 r6:60030193 r5:00000000 r4:c0d7e300 [ 201.977519] [<c010ef44>] (show_stack) from [<c0859d80>] (dump_stack+0xb4/0xec) [ 201.984760] [<c0859ccc>] (dump_stack) from [<c017bf80>] (print_circular_bug.constprop.15+0x264/0x328) [ 201.993996] r10:c14e4e24 r9:00000000 r8:c8094b00 r7:c8095038 r6:c1200718 r5:c11e2e38 [ 202.001836] r4:c1200718 r3:14127a92 [ 202.005430] [<c017bd1c>] (print_circular_bug.constprop.15) from [<c017edec>] (__lock_acquire+0x14a8/0x1970) [ 202.015186] r10:00000002 r9:c1491038 r8:c8094b00 r7:c1200718 r6:c8094ff8 r5:00000001 [ 202.023027] r4:c1200718 r3:c8094ff8 [ 202.026620] [<c017d944>] (__lock_acquire) from [<c017fc04>] (lock_acquire+0xd0/0x1f4) [ 202.034466] r10:c0d08930 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c8ccd5ec [ 202.042303] r4:60030193 [ 202.044856] [<c017fb34>] (lock_acquire) from [<c0877fcc>] (_raw_spin_lock_irqsave+0x40/0x54) [ 202.053310] r10:00000241 r9:00000015 r8:000000e3 r7:00030193 r6:c04d434c r5:60030193 [ 202.061148] r4:c8ccd5dc [ 202.063704] [<c0877f8c>] (_raw_spin_lock_irqsave) from [<c04d434c>] (gsmld_write_wakeup+0x28/0x5c) [ 202.072674] r6:00000fff r5:c8ccd5dc r4:c8ccd400 [ 202.077313] [<c04d4324>] (gsmld_write_wakeup) from [<c04c46a8>] (tty_wakeup+0x58/0x64) [ 202.085244] r7:00030193 r6:00000fff r5:c8d759c0 r4:c8ccdc00 [ 202.090922] [<c04c4650>] (tty_wakeup) from [<c04cf0b0>] (tty_port_default_wakeup+0x1c/0x28) [ 202.099284] r5:c82f0000 r4:c8ccdc00 [ 202.102875] [<c04cf094>] (tty_port_default_wakeup) from [<c04cec68>] (tty_port_tty_wakeup+0x18/0x1c) [ 202.112018] r5:c82f0000 r4:c8322440 [ 202.115610] [<c04cec50>] (tty_port_tty_wakeup) from [<c04e9a88>] (uart_write_wakeup+0x1c/0x24) [ 202.124239] [<c04e9a6c>] (uart_write_wakeup) from [<c04ee3d4>] (imx_uart_txint+0x13c/0x198) [ 202.132605] [<c04ee298>] (imx_uart_txint) from [<c04ee530>] (imx_uart_int+0x100/0x1a4) [ 202.140538] r9:00000015 r8:00000000 r7:00000000 r6:00000000 r5:c8322440 r4:00002040 [ 202.148299] [<c04ee430>] (imx_uart_int) from [<c018df68>] (__handle_irq_event_percpu+0x44/0x364) [ 202.157100] r10:00000015 r9:c80b3da8 r8:00000001 r7:00000000 r6:c0d08930 r5:c8158464 [ 202.164941] r4:c8343e40 r3:c04ee430 [ 202.168534] [<c018df24>] (__handle_irq_event_percpu) from [<c018e2b8>] (handle_irq_event_percpu+0x30/0x88) [ 202.178203] r10:c0d42038 r9:c8010400 r8:00000001 r7:00000000 r6:c8158400 r5:c8158464 [ 202.186042] r4:c0d08908 [ 202.188594] [<c018e288>] (handle_irq_event_percpu) from [<c018e350>] (handle_irq_event+0x40/0x64) [ 202.197477] r6:c0d14f0c r5:c8158464 r4:c8158400 [ 202.202114] [<c018e310>] (handle_irq_event) from [<c019201c>] (handle_fasteoi_irq+0xc4/0x174) [ 202.210652] r7:00000000 r6:c0d14f0c r5:c8158464 r4:c8158400 [ 202.216330] [<c0191f58>] (handle_fasteoi_irq) from [<c018d090>] (generic_handle_irq+0x28/0x3c) [ 202.224956] r7:00000000 r6:c0d08d08 r5:00000015 r4:c0c8d128 [ 202.230633] [<c018d068>] (generic_handle_irq) from [<c018d71c>] (__handle_domain_irq+0x6c/0xe8) [ 202.239350] [<c018d6b0>] (__handle_domain_irq) from [<c048a358>] (gic_handle_irq+0x60/0xc4) [ 202.247717] r9:c80b3e80 r8:c0d08f34 r7:d080a000 r6:000003ff r5:000003eb r4:d080a00c [ 202.255478] [<c048a2f8>] (gic_handle_irq) from [<c0101a30>] (__irq_svc+0x70/0x98) [ 202.262972] Exception stack(0xc80b3e80 to 0xc80b3ec8) [ 202.268043] 3e80: 00000001 00000001 00000000 0000b51a c8010000 c8028900 c8010000 c800f000 [ 202.276238] 3ea0: c80b3f04 c82f0418 c8010000 c80b3ee4 00000000 c80b3ed0 c017fe38 c087816c [ 202.284425] 3ec0: 20030113 ffffffff [ 202.287930] r10:c8010000 r9:c80b2000 r8:c80b3f04 r7:c80b3eb4 r6:ffffffff r5:20030113 [ 202.295769] r4:c087816c [ 202.298323] [<c087813c>] (_raw_spin_unlock_irq) from [<c014728c>] (process_one_work+0x210/0x718) [ 202.307122] r5:c8028900 r4:c82f0414 [ 202.310716] [<c014707c>] (process_one_work) from [<c01477c0>] (worker_thread+0x2c/0x54c) [ 202.318826] r10:c8010000 r9:c0d05900 r8:00000088 r7:c8010034 r6:c8010000 r5:c8028914 [ 202.326666] r4:c8028900 [ 202.329216] [<c0147794>] (worker_thread) from [<c014e6c4>] (kthread+0x14c/0x164) [ 202.336628] r10:c8099e40 r9:c0147794 r8:c8028900 r7:c80b2000 r6:c8037fc0 r5:c8028980 [ 202.344468] r4:00000000 r3:00000000 [ 202.348060] [<c014e578>] (kthread) from [<c01010b4>] (ret_from_fork+0x14/0x20) [ 202.355292] Exception stack(0xc80b3fb0 to 0xc80b3ff8) [ 202.360356] 3fa0: 00000000 00000000 00000000 00000000 [ 202.368549] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 202.376740] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 [ 202.383369] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c014e578 [ 202.391208] r4:c8037fc0
| |