lkml.org 
[lkml]   [2021]   [Jan]   [28]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: [PATCH v2] nvme-multipath: Early exit if no path is available
From
Date


On 2021/1/28 17:40, Daniel Wagner wrote:
> On Thu, Jan 28, 2021 at 05:18:56PM +0800, Chao Leng wrote:
>> It is impossible to return NULL for nvme_next_ns(head, old).
>
> block nvme0n1: no usable path - requeuing I/O
> block nvme0n1: no usable path - requeuing I/O
> block nvme0n1: no usable path - requeuing I/O
> block nvme0n1: no usable path - requeuing I/O
> block nvme0n1: no usable path - requeuing I/O
> block nvme0n1: no usable path - requeuing I/O
> BUG: kernel NULL pointer dereference, address: 0000000000000068
> #PF: supervisor read access in kernel mode
> #PF: error_code(0x0000) - not-present page
> PGD 8000000ff67bc067 P4D 8000000ff67bc067 PUD ff9ac9067 PMD 0
> Oops: 0000 [#1] SMP PTI
> CPU: 23 PID: 15759 Comm: dt.21.15 Kdump: loaded Tainted: G E 5.3.18-0.gc9fe679-default #1 SLE15-SP2 (unreleased)
> Hardware name: FUJITSU PRIMERGY RX2540 M2/D3289-B1, BIOS V5.0.0.11 R1.18.0 for D3289-B1x 02/06/2018
> RIP: 0010:nvme_ns_head_make_request+0x1d1/0x430 [nvme_core]
> Code: 54 24 10 0f 84 c9 01 00 00 48 8b 54 24 10 48 83 ea 30 0f 84 ba 01 00 00 48 39 d0 0f 84 01 02 00 00 31 ff eb 05 48 39 d0 74 67 <48> 8b 72 68 83 e6 04 75 13 48 8b 72 68 83 e6 01 75 0a 48 8b 72 10
> RSP: 0018:ffffa69d08017af8 EFLAGS: 00010246
> RAX: ffff92f261d87800 RBX: ffff92fa555b0010 RCX: ffff92fa555bc570
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> RBP: 0000000000000001 R08: 0000000000001000 R09: 0000000000001000
> R10: ffffa69d080179a8 R11: ffff92f264f0c1c0 R12: ffff92f264f7f000
> R13: ffff92fa555b0000 R14: 0000000000000001 R15: 0000000000000000
> FS: 00007f3962bae700(0000) GS:ffff92f29ffc0000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000068 CR3: 0000000fd69a2002 CR4: 00000000003606e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
> generic_make_request+0x121/0x300
> ? submit_bio+0x42/0x1c0
> submit_bio+0x42/0x1c0
> ext4_io_submit+0x49/0x60 [ext4]
> ext4_writepages+0x625/0xe90 [ext4]
> ? do_writepages+0x4b/0xe0
> ? ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4]
> do_writepages+0x4b/0xe0
> ? __generic_file_write_iter+0x192/0x1c0
> ? __filemap_fdatawrite_range+0xcb/0x100
> __filemap_fdatawrite_range+0xcb/0x100
> ? ext4_file_write_iter+0x128/0x3c0 [ext4]
> file_write_and_wait_range+0x5e/0xb0
> __generic_file_fsync+0x22/0xb0
> ext4_sync_file+0x1f7/0x3c0 [ext4]
> do_fsync+0x38/0x60
> __x64_sys_fsync+0x10/0x20
> do_syscall_64+0x5b/0x1e0
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> You can't see exactly where it dies but I followed the assembly to
> nvme_round_robin_path(). Maybe it's not the initial nvme_next_ns(head,
> old) which returns NULL but nvme_next_ns() is returning NULL eventually
> (list_next_or_null_rcu()).
So there is other bug cause nvme_next_ns abormal.
I review the code about head->list and head->current_path, I find 2 bugs
may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/20210128033351.22116-1-lengchao@huawei.com/
Second, in nvme_ns_remove, list_del_rcu is before
nvme_mpath_clear_current_path. This may cause "old" is deleted from the
"head", but still use "old". I'm not sure there's any other
consideration here, I will check it and try to fix it.
>
> And I have positive feedback, this patch fixes the above problem.
Although adding check ns can fix the crash. There may still be other problems
such as in __nvme_find_path which use list_for_each_entry_rcu.
>
> .
>

\
 
 \ /
  Last update: 2021-01-29 02:26    [W:0.074 / U:0.108 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site