lkml.org 
[lkml]   [2022]   [Jun]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRaid5 Reshape Data Corruption Bug

Hey,

I've diagnosed a bug in the reshape code that corrupts data, however I
don't have a good solution to the problem and a solution may be quite
complicated. I suspect this is the cause of random failures I see with
01r5integ and 01raid6integ. (Though I can't say for certain as I have a
quicker reproduction method.)

The bug is that during reshape: EXPAND_SOURCE stripes are not in the
correct order when the data comes back from the disk and if a latter
stripe comes back sooner than an earlier EXPAND_SOURCE stripe has read
the disk then an EXPAND_READY stripe might write a block before the
source block was able to read it; so it overwrites data before that data
has been moved and result in corrupt data on the disk. This happens
reasonably frequently.

I suspect this is made worse with modern SSDs and spinning disks would
be less likely to exhibit this problem as it would naturally try to
order the reads by sector.

So somehow there needs to be a way to prevent an EXPAND_READY stripe
from writing the data for a specific device block before the
corresponding EXPAND_SOURCE block has read the data. And I don't see an
trivial way to get that done.

If anyone has any clever solutions it would be good to hear that.
Otherwise, I don't think I'll have time to find a solution myself.

Logan

\
 
 \ /
  Last update: 2022-06-11 00:10    [W:0.029 / U:0.228 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site