Messages in this thread | | | From | rh <> | Subject | Re: usb storage: FATAL DATA CORRUPTION due to innocuous reboot!? | Date | Wed, 30 Jan 2013 14:29:48 -0800 |
| |
On Sat, 26 Jan 2013 14:22:32 +0100 Andreas Mohr <andi@lisas.de> wrote:
> Hi, > > after my recent tirade of very poor device support of Aspire One, > I now experienced something a lot worse (bad karma? ;-P): > basically my entire ext4 root partition got blewn into shreds > (corruption is so pervasive that I'm afraid recovery will fail). > > I am (was) running 3.7.0, and decided to upgrade to current (-rc4+). > Thus I did grub-mkconfig and more or less immediately rebooted. > Realized that I had failed to copy vmlinuz to /boot's bzImage > (i.e. new boot entry was missing), rebooted and redid that, > re-ran grub-mkconfig and rebooted more or less immediately. > > After the first grub-mkconfig GRUB2 was still fine, > being able to boot the existing kernel. > Exactly directly after the second reboot (post-grub-mkconfig) > all hell broke lose, with GRUB2 complaining about "invalid extent" > and a subsequent fsck.ext4 spewing tons of pages of errors. > > I'm using the infamous JMF601 SSD controller, USB-connected > (root device). > Cannot provide details of grub package version since root partition > is toast. > > Note that the first inode that fsck complained about was 262144, i.e. > 0x40000 i.e. 256kB i.e. most certainly directly at a boundary of > erase block size. IOW, the corruption is very likely to have been > produced by coarse erase block related activity and *not* by any > interim merging of *partial* data updates. > > > While of course the corruption may have happened due to a questionable > device, I now have a hunch that this unspeakable mess has been caused > by the reboot happening too early (while the SSD was still writing > data, probably by having to actively and painfully erase formerly > used blocks, too). If the reboot happened too early, this would > probably mean that USB port power during reboot got lost too early, > thus the controller lost power during ongoing data updates. > If the controller's operation happens to be implemented > in a not fully atomic way (as is somewhat likely given JMF601's > reputation), then this means data corruption, plenty. > > Thus I started to investigate about the kernel's device consistency > guarantees upon reboot. > Note that reboot(8) says: > " > The -h flag puts all hard disks in standby mode just before > halt or > power-off. Right now this is only implemented for IDE > drives. A side > effect of putting the drive in stand-by mode is that the write > cache on > the disk is flushed. This is important for IDE drives, since > the kernel > doesn't flush the write cache itself before power-off. > " > Excuse me!? > Why wouldn't the kernel be responsible to take care to flush things > prior to power-off? > > Also, > http://linux.die.net/man/8/sync > (a possibly old/irrelevant source) says: > "The reboot(8) and halt(8) commands take this into account by sleeping > for a few seconds after calling sync(2)" > > Please forgive me for a second that I'm *very* puzzled why > it would be the reboot binary's job to do a delay to ensure > properly completed syncing/flushing of the storage devices. > After all it's quite arguably definitely the *kernel*'s job > to govern device-specific flush delay requirements (only the kernel > knows which particular device may have certain particularly special > manual delay requirements, and all that a userspace binary ought > to do is to issue a *client request* for a reboot). > > > Please note that the sync binary is only about syncing > filesystem-related parts, i.e. it does NOT seem to be responsible for > the (much more important!!) non-fs parts such as the things updated > by GRUB (is this the hole that I'm seeing here?). > > So, to have a short list: > - I suspect improper sync/flush handling prior to reboot > (which likely eventually leads to the obviously fatal USB port > poweroff) > - it's possibly the case that sync handling is sufficient to handle > FS parts but not the even more critical non-FS parts (bootloader) > - kernel might actually do a proper sync/flush of *all* device parts, > but device may fail to obey it > > If it in fact is a problem specific to this device, > then it might be conceivable to introduce a new USB storage quirk flag > for devices with broken flush which would add an arbitrary pre-reboot > delay of perhaps 10 seconds. > If the kernel has a last-write-at timestamp per block device > (which it arguably should maintain), then this could be used to > shorten the delay to the time remaining since last write > (which also would allow to prolong the total delay to 20 seconds). > > Questions: > - should I file a kernel bug report about this issue? > - did anyone experience anything similar? (research didn't > manage to locate much so far) > - if my thoughts are correct (about storage quirk), how to implement > it? > - any other hints/ideas?
I am curious to know if you found anything more about this.
I have seen some strange things lately with USB devices. A 64MB (yes MB) SD card was not readable on linux 3.6.7 but was fine on macos. The card was created under a previous linux kernel. I also saw flushing issues on other usb mem sticks.
1. copied a large file from disk -> usb stick 2. cp finished 3. I umount'd the usb stick 4. put stick in other pc 5. file was truncated
Had to redo those steps but also did a bin/sync before unmounting.
> > > I have to admit that all these way too many kernel "features" > are really adding up going on my nerves (Alan Cox, anyone?). > If this keeps going on, then I *will* be forced to bail out, hard.
I have been looking at alternatives to linux kernel. Is that what you mean?
> > Thanks, > > Andreas Mohr > -- > To unsubscribe from this list: send the line "unsubscribe linux-usb" > in the body of a message to > majordomo@vger.kernel.org More majordomo info > at http://vger.kernel.org/majordomo-info.html >
--
| |