[lkml]   [2008]   [Jun]   [23]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Subjectext3 directory corruption under Xen
We've been seeing a rash of ext3 directory corruption occurring under 
Xen. All but one of the reports have been with filesystems formatted
with 1024 blocksize. We have one report, that's potentialy the same
bug, occurring on a filesystem with 4096 blocksize (either way, it was
some type of corruption in that case). In all cases, the filesystems
were mounted with ext3's default journaling mode. No quotas or anything
else other than the default ext3 mount options.

It's happened on a number of different hosts, all of the same hardware
and software configuration (Xen 3.2 64bit, 32bit pae dom0, 32bit pae
domUs. LVM backend with 3ware hardware RAID-1). Some of those hosts
were previously running non-virtaulized Linux and UML, using the
identical guest images, and under that configuration never experienced
this problem.

This has occurred under both 2.6.18-xenbits and the more recent pv_ops
based kernels (2.6.24, 2.6.25), which I presume are all using the same
blkfront driver code.

The common workloads from the reports seems to be active maildirs and rsync.

The initial errors reported back are all from fs/ext3/dir.c, in
ext3_check_dir_entry(). Most commonly hit is the "rec_len % 4 != 0"
check. We've seen other checks trigger, but my assumption is that those
happen after more stuff gets whacked out.

Eventually the fs will go read-only. In extreme cases, the fs is chewed
through enough that data is lost.

It's tricky to track down the trigger because you can only detect the
corruption after it's happened. Our attempts to reproduce this using
various filesystem thrashing scripts haven't yielded a reliable way to
trigger it, however we have been successful in triggering it twice -- in
two weeks :( .

My hope is that this triggers an "a-hah" from someone in LKML or Xen
land who has experience with this code, or that this is a known issue
and a fix already lives.

We're scared. Please help.


 \ /
  Last update: 2008-06-23 18:33    [W:0.046 / U:1.960 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site