Messages in this thread | | | From | "Pradeep Pragallapati (QUIC)" <> | Subject | RE: [PATCH V1] block: Fix null pointer dereference issue on struct io_cq | Date | Thu, 18 May 2023 09:25:52 +0000 |
| |
Hi,
-----Original Message----- From: Damien Le Moal <dlemoal@kernel.org> Sent: Wednesday, May 17, 2023 3:02 PM To: Christoph Hellwig <hch@infradead.org> Cc: Yu Kuai <yukuai1@huaweicloud.com>; Pradeep Pragallapati (QUIC) <quic_pragalla@quicinc.com>; axboe@kernel.dk; linux-block@vger.kernel.org; linux-kernel@vger.kernel.org; yukuai (C) <yukuai3@huawei.com> Subject: Re: [PATCH V1] block: Fix null pointer dereference issue on struct io_cq
On 5/17/23 18:21, Christoph Hellwig wrote: > On Wed, May 17, 2023 at 06:20:19PM +0900, Damien Le Moal wrote: >> twice for the same icq. The missing rcu lock in ioc_exit_icqs() >> already was in itself a bug, and the missing flag check is another. > > spinlocks imply a rcu critical section, no need to duplicate it.
Right. And I misread the code. As Yu said, given that ioc_exit_icqs() iterates the list of icqs under ioc->lock and the ioc is removed from that list under the same lock, ioc_exit_icqs() should never see an icq that went through ioc_destroy_icq()... Very weird.
This weird can be possible 1. updating icq_hint which is annotated as __rcu type without RCU-protected context in ioc_destroy_icq(). Moreover, this was taken care in else part of ioc_release_fn() by rcu_read_lock/unlock() but missed in if statement which can lead to this weird.
2. extracting icq from hlist/list elements are done using rcu locks protected in ioc_clear_queue() but same was not at ioc_exit_icqs().
So, far we have seen 10+ instances of this crash on 6.1 kernel during stability testing (Involves IO, reboots, device suspend/resume, and few more). With the V1 patch, we didn't observe the issue for at least 48hrs+ of stability testing.
-- Damien Le Moal Western Digital Research
| |