Messages in this thread | | | Date | Mon, 11 Apr 2022 16:03:40 -0500 | Subject | Re: [PATCH v5] locking/rwsem: Make handoff bit handling more consistent | From | john.p.donnelly@oracle ... |
| |
On 4/11/22 1:40 PM, Waiman Long wrote: > On 4/11/22 14:26, john.p.donnelly@oracle.com wrote: >> On 2/14/22 9:47 AM, chenguanyou wrote: >>> Hi Waiman, Greg, >>> This patch has been merged in branch linux-5.16.y. >>> Can we take it to the linux-5.10.y LTS version? >>> >>> thanks, >> >> Hi, >> >> As a FYI: >> >> We have observed that following lockup with this commit added to >> 5.15.LTS: >> >> d257cc8cb8d5 - locking/rwsem: Make handoff bit handling more >> consistent (4 months ago) <Waiman Long> >> >> The "fio" test suit fails with LVM devices composed of four NVME >> devices with these observed lockup, panic. >> >> >> >> ext4: >> >> PID: 3682 TASK: ffff8f489ae34bc0 CPU: 2 COMMAND: "dio/dm-0" >> #0 [fffffe0000083e50] crash_nmi_callback at ffffffff828772b3 >> #1 [fffffe0000083e58] nmi_handle at ffffffff82840778 >> #2 [fffffe0000083ea0] default_do_nmi at ffffffff8337a1e2 >> #3 [fffffe0000083ec8] exc_nmi at ffffffff8337a48d >> #4 [fffffe0000083ef0] end_repeat_nmi at ffffffff8340153b >> [exception RIP: _raw_spin_lock_irq+23] >> RIP: ffffffff8338b2e7 RSP: ffff9c4409b47c78 RFLAGS: 00000046 >> RAX: 0000000000000000 RBX: ffff8f489ae34bc0 RCX: 0000000000000000 >> RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8f47f7b90104 >> RBP: ffff9c4409b47d20 R8: 0000000000000000 R9: 0000000000000000 >> R10: 0000000000000000 R11: 0000000000000000 R12: ffff8f47f7b90104 >> R13: ffff9c4409b47cb0 R14: ffff8f47f7b900f0 R15: 0000000000000000 >> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 >> <NMI exception stack> --- >> #5 [ffff9c4409b47c78] _raw_spin_lock_irq at ffffffff8338b2e7 >> #6 [ffff9c4409b47c78] rwsem_down_write_slowpath at ffffffff82925be9 >> #7 [ffff9c4409b47d28] ext4_map_blocks at ffffffffc11c26dc [ext4] >> #8 [ffff9c4409b47d98] ext4_convert_unwritten_extents at >> ffffffffc11ad9e0 [ext4] >> #9 [ffff9c4409b47df0] ext4_dio_write_end_io at ffffffffc11b22aa [ext4] >> >> xfs: >> >> PID: 3719 TASK: ffff9f81d2d74bc0 CPU: 37 COMMAND: "dio/dm-0" >> #0 [fffffe0000894e50] crash_nmi_callback at ffffffffad6772b3 >> #1 [fffffe0000894e58] nmi_handle at ffffffffad640778 >> #2 [fffffe0000894ea0] default_do_nmi at ffffffffae17a1e2 >> #3 [fffffe0000894ec8] exc_nmi at ffffffffae17a48d >> #4 [fffffe0000894ef0] end_repeat_nmi at ffffffffae20153b >> [exception RIP: _raw_spin_lock_irq+23] >> RIP: ffffffffae18b2e7 RSP: ffffbb7ec9637c48 RFLAGS: 00000046 >> RAX: 0000000000000000 RBX: ffff9f81d2d74bc0 RCX: 0000000000000000 >> RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9f81c04a918c >> RBP: ffffbb7ec9637ce8 R8: 0000000000000000 R9: 0000000000000000 >> R10: 0000000000000000 R11: 0000000000000000 R12: ffff9f81c04a918c >> R13: ffffbb7ec9637c80 R14: ffff9f81c04a9178 R15: 0000000000000000 >> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 >> <NMI exception stack> --- >> #5 [ffffbb7ec9637c48] _raw_spin_lock_irq at ffffffffae18b2e7 >> #6 [ffffbb7ec9637c48] rwsem_down_write_slowpath at ffffffffad725be9 >> #7 [ffffbb7ec9637cf0] xfs_trans_alloc_inode at ffffffffc074f2bd [xfs] >> #8 [ffffbb7ec9637d50] xfs_iomap_write_unwritten at ffffffffc073ad15 >> [xfs] >> #9 [ffffbb7ec9637dd0] xfs_dio_write_end_io at ffffffffc072db62 [xfs] >> >> >> I have reached out to Waiman and he suggested this for our next test >> pass: >> >> >> 1ee326196c6658 locking/rwsem: Always try to wake waiters in out_nolock >> path > > Does this commit help to avoid the lockup problem? > > Commit 1ee326196c6658 fixes a potential missed wakeup problem when a > reader first in the wait queue is interrupted out without acquiring the > lock. It is actually not a fix for commit d257cc8cb8d5. However, this > commit changes the out_nolock path behavior of writers by leaving the > handoff bit set when the wait queue isn't empty. That likely makes the > missed wakeup problem easier to reproduce. > > Cheers, > Longman >
Hi,
We are testing now
ETA for fio soak test completion is ~15hr from now.
I wanted to share the stack traces for future reference + occurrences.
Cheers.
JD
...
| |