lkml.org 
[lkml]   [2022]   [Jan]   [24]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    From
    Subject[PATCH 5.15 710/846] btrfs: fix deadlock between quota enable and other quota operations
    Date
    From: Filipe Manana <fdmanana@suse.com>

    commit 232796df8c1437c41d308d161007f0715bac0a54 upstream.

    When enabling quotas, we attempt to commit a transaction while holding the
    mutex fs_info->qgroup_ioctl_lock. This can result on a deadlock with other
    quota operations such as:

    - qgroup creation and deletion, ioctl BTRFS_IOC_QGROUP_CREATE;

    - adding and removing qgroup relations, ioctl BTRFS_IOC_QGROUP_ASSIGN.

    This is because these operations join a transaction and after that they
    attempt to lock the mutex fs_info->qgroup_ioctl_lock. Acquiring that mutex
    after joining or starting a transaction is a pattern followed everywhere
    in qgroups, so the quota enablement operation is the one at fault here,
    and should not commit a transaction while holding that mutex.

    Fix this by making the transaction commit while not holding the mutex.
    We are safe from two concurrent tasks trying to enable quotas because
    we are serialized by the rw semaphore fs_info->subvol_sem at
    btrfs_ioctl_quota_ctl(), which is the only call site for enabling
    quotas.

    When this deadlock happens, it produces a trace like the following:

    INFO: task syz-executor:25604 blocked for more than 143 seconds.
    Not tainted 5.15.0-rc6 #4
    "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    task:syz-executor state:D stack:24800 pid:25604 ppid: 24873 flags:0x00004004
    Call Trace:
    context_switch kernel/sched/core.c:4940 [inline]
    __schedule+0xcd9/0x2530 kernel/sched/core.c:6287
    schedule+0xd3/0x270 kernel/sched/core.c:6366
    btrfs_commit_transaction+0x994/0x2e90 fs/btrfs/transaction.c:2201
    btrfs_quota_enable+0x95c/0x1790 fs/btrfs/qgroup.c:1120
    btrfs_ioctl_quota_ctl fs/btrfs/ioctl.c:4229 [inline]
    btrfs_ioctl+0x637e/0x7b70 fs/btrfs/ioctl.c:5010
    vfs_ioctl fs/ioctl.c:51 [inline]
    __do_sys_ioctl fs/ioctl.c:874 [inline]
    __se_sys_ioctl fs/ioctl.c:860 [inline]
    __x64_sys_ioctl+0x193/0x200 fs/ioctl.c:860
    do_syscall_x64 arch/x86/entry/common.c:50 [inline]
    do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
    entry_SYSCALL_64_after_hwframe+0x44/0xae
    RIP: 0033:0x7f86920b2c4d
    RSP: 002b:00007f868f61ac58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
    RAX: ffffffffffffffda RBX: 00007f86921d90a0 RCX: 00007f86920b2c4d
    RDX: 0000000020005e40 RSI: 00000000c0109428 RDI: 0000000000000008
    RBP: 00007f869212bd80 R08: 0000000000000000 R09: 0000000000000000
    R10: 0000000000000000 R11: 0000000000000246 R12: 00007f86921d90a0
    R13: 00007fff6d233e4f R14: 00007fff6d233ff0 R15: 00007f868f61adc0
    INFO: task syz-executor:25628 blocked for more than 143 seconds.
    Not tainted 5.15.0-rc6 #4
    "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    task:syz-executor state:D stack:29080 pid:25628 ppid: 24873 flags:0x00004004
    Call Trace:
    context_switch kernel/sched/core.c:4940 [inline]
    __schedule+0xcd9/0x2530 kernel/sched/core.c:6287
    schedule+0xd3/0x270 kernel/sched/core.c:6366
    schedule_preempt_disabled+0xf/0x20 kernel/sched/core.c:6425
    __mutex_lock_common kernel/locking/mutex.c:669 [inline]
    __mutex_lock+0xc96/0x1680 kernel/locking/mutex.c:729
    btrfs_remove_qgroup+0xb7/0x7d0 fs/btrfs/qgroup.c:1548
    btrfs_ioctl_qgroup_create fs/btrfs/ioctl.c:4333 [inline]
    btrfs_ioctl+0x683c/0x7b70 fs/btrfs/ioctl.c:5014
    vfs_ioctl fs/ioctl.c:51 [inline]
    __do_sys_ioctl fs/ioctl.c:874 [inline]
    __se_sys_ioctl fs/ioctl.c:860 [inline]
    __x64_sys_ioctl+0x193/0x200 fs/ioctl.c:860
    do_syscall_x64 arch/x86/entry/common.c:50 [inline]
    do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
    entry_SYSCALL_64_after_hwframe+0x44/0xae

    Reported-by: Hao Sun <sunhao.th@gmail.com>
    Link: https://lore.kernel.org/linux-btrfs/CACkBjsZQF19bQ1C6=yetF3BvL10OSORpFUcWXTP6HErshDB4dQ@mail.gmail.com/
    Fixes: 340f1aa27f36 ("btrfs: qgroups: Move transaction management inside btrfs_quota_enable/disable")
    CC: stable@vger.kernel.org # 4.19
    Reviewed-by: Qu Wenruo <wqu@suse.com>
    Signed-off-by: Filipe Manana <fdmanana@suse.com>
    Signed-off-by: David Sterba <dsterba@suse.com>
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
    ---
    fs/btrfs/qgroup.c | 19 +++++++++++++++++++
    1 file changed, 19 insertions(+)

    --- a/fs/btrfs/qgroup.c
    +++ b/fs/btrfs/qgroup.c
    @@ -940,6 +940,14 @@ int btrfs_quota_enable(struct btrfs_fs_i
    int ret = 0;
    int slot;

    + /*
    + * We need to have subvol_sem write locked, to prevent races between
    + * concurrent tasks trying to enable quotas, because we will unlock
    + * and relock qgroup_ioctl_lock before setting fs_info->quota_root
    + * and before setting BTRFS_FS_QUOTA_ENABLED.
    + */
    + lockdep_assert_held_write(&fs_info->subvol_sem);
    +
    mutex_lock(&fs_info->qgroup_ioctl_lock);
    if (fs_info->quota_root)
    goto out;
    @@ -1117,8 +1125,19 @@ out_add_root:
    goto out_free_path;
    }

    + mutex_unlock(&fs_info->qgroup_ioctl_lock);
    + /*
    + * Commit the transaction while not holding qgroup_ioctl_lock, to avoid
    + * a deadlock with tasks concurrently doing other qgroup operations, such
    + * adding/removing qgroups or adding/deleting qgroup relations for example,
    + * because all qgroup operations first start or join a transaction and then
    + * lock the qgroup_ioctl_lock mutex.
    + * We are safe from a concurrent task trying to enable quotas, by calling
    + * this function, since we are serialized by fs_info->subvol_sem.
    + */
    ret = btrfs_commit_transaction(trans);
    trans = NULL;
    + mutex_lock(&fs_info->qgroup_ioctl_lock);
    if (ret)
    goto out_free_path;


    \
     
     \ /
      Last update: 2022-01-25 01:13    [W:2.937 / U:0.852 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site