Messages in this thread | | | Date | Sat, 01 Nov 2008 16:08:36 +0300 | From | Vladislav Bolkhovitin <> | Subject | Re: FS corruption after I/O errors |
| |
Vladislav Bolkhovitin wrote: > Nick Piggin wrote: >> On Wednesday 29 October 2008 06:38, Vladislav Bolkhovitin wrote: >>> Nick Piggin wrote: >>>> On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote: >>>>> Hi, >>>>> >>>>> During recent debugging session of my SCSI target SCST >>>>> (http://scst.sf.net) I noticed many >>>>> >>>>> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66() >>>>> >>>>> messages in kernel log on the initiator. I attached the full log of >>>>> several of them. >>>>> >>>>> My target was buggy and I was working on fixing it, but I suppose Linux >>>>> should handle such failures more gracefully. In all the cases the target >>>>> had one type of failure: it "ate" a SCSI command and never returned >>>>> result of it. >>>> Right. This is one of the warnings I see in my fault-injection testing. >>>> It is fixed by my patch to clean up and improve the page and buffer >>>> error handling in the vm/fs. >>> Can you specify which patch you referring? Is it in 2.6.27? >> It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27. > > I see. I'm looking forward to see it in 2.6.28 or .29. This is really a > needed work. > > BTW, have you even seen in your fault-injection testing that after > receiving a failure from a SCSI device during heavy load ext3 file > system mounted on it gets corrupted and journal replay on remount > doesn't repair it, only manual e2fsck helps? I've many times seen that, > including cases when the target was remaining up and fully functional. > See, e.g., "MOANING MODE ON" part in > http://marc.info/?l=linux-scsi&m=121932252324432&w=2. I haven't checked > that case since then, although I see such corruptions quite often. But > in all them I can't so clearly say that it isn't a target's failure.
I've just checked it with 2.6.27. The situation greatly improved and dbench was able to complete several runs under constant TASK_ABORTED "bombarding" (TASK RESET task management commands using "sg_reset -b" each 31 seconds from another "connection" to that device via qla2xxx initiator driver. You can see those resets in the attached log). But when then I unmounted the affected partition, e2fsck found errors on it. See attachments for details. The target all the times was fine and completely healthy.
[root@inif w]# e2fsck -f -y /dev/sdc e2fsck 1.38 (30-Jun-2005) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/sdc: 69413/640000 files (0.2% non-contiguous), 316508/1280000 blocks [root@inif w]# mount -t ext3 /dev/sdc /mnt [root@inif dbench-mod]# while true; do ./dbench 50; done 50 clients started .........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+...................................+..........................................................................+.................................................................+.+.................................................+.....................................+................................................................................................................++...........................++....................++..........................+.......................+......................................+.................................................................................................+.........................++++.+...+................................+...++.+........++..........+.................++++......................+++....+.++.....+.....+.....................+......+++.++++************************************************** Throughput 27.5109 MB/sec (NB=34.3887 MB/sec 275.109 MBit/sec) Average latency 7.166755 ms Maximum latency 22972.843000 ms Latency > 2 sec. 0.091621 % Latency > 10 sec. 0.016009 % 50 clients started ............................................................................................................write failed on handle 4290 ..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+............................................................................................................................................................................................................................................................................................................................................................................................................................++.........+......................................write failed on handle 4279 ........................+.................................................+...................................................+...............+.........+...............................+.............++..............................++...........................+++........................+...+............+......+...+.+......+.....+...............++.....+....................+.....+..+++.+............+....+++......+.+...++.....+........+........+.+++++************************************************** Throughput 27.5921 MB/sec (NB=34.4902 MB/sec 275.921 MBit/sec) Average latency 7.097532 ms Maximum latency 20892.153000 ms Latency > 2 sec. 0.087833 % Latency > 10 sec. 0.016437 % 50 clients started ....................................................................................write failed on handle 4290 ..(4727) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory (4729) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4328 (File exists) (4730) nb_close: handle 4328 was not open (4739) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory (4743) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory (4745) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4329 (File exists) (4751) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4330 (File exists) (4752) nb_read: handle 4330 was not open size=24 ofs=2156 (4754) nb_read: handle 4330 was not open size=276 ofs=2139 (4755) nb_read: handle 4330 was not open size=14 ofs=2136 (4757) nb_read: handle 4330 was not open size=7 ofs=2136 (4759) nb_read: handle 4330 was not open size=12 ofs=2138 (4761) nb_read: handle 4330 was not open size=8 ofs=2137 (4763) nb_read: handle 4330 was not open size=15 ofs=2135 (4765) nb_read: handle 4330 was not open size=16 ofs=0 (4767) nb_read: handle 4330 was not open size=2048 ofs=16 (4768) nb_read: handle 4330 was not open size=2048 ofs=2064 (4769) nb_read: handle 4330 was not open size=2048 ofs=257 (4770) nb_read: handle 4330 was not open size=2048 ofs=385 (4771) nb_read: handle 4330 was not open size=2048 ofs=513 (4772) nb_read: handle 4330 was not open size=2048 ofs=641 (4773) nb_read: handle 4330 was not open size=2048 ofs=769 (4774) nb_read: handle 4330 was not open size=2048 ofs=897 (4775) nb_read: handle 4330 was not open size=2048 ofs=1025 (4776) nb_read: handle 4330 was not open size=2048 ofs=1153 (4777) nb_read: handle 4330 was not open size=2048 ofs=1281 (4778) nb_read: handle 4330 was not open size=2048 ofs=1409 (4779) nb_read: handle 4330 was not open size=408 ofs=1537 ..(4899) nb_close: handle 4330 was not open (4964) nb_read: handle 4329 was not open size=1386 ofs=2157 (4971) nb_read: handle 4329 was not open size=4096 ofs=2157 (4972) nb_read: handle 4329 was not open size=4096 ofs=6253 (4973) nb_read: handle 4329 was not open size=4096 ofs=10349 (4974) nb_read: handle 4329 was not open size=4096 ofs=14445 (4975) nb_read: handle 4329 was not open size=4096 ofs=18541 (4976) nb_read: handle 4329 was not open size=256 ofs=22637 (4977) nb_read: handle 4329 was not open size=4096 ofs=22893 (4978) nb_read: handle 4329 was not open size=4096 ofs=26989 (4979) nb_read: handle 4329 was not open size=4096 ofs=31085 (4980) nb_read: handle 4329 was not open size=4096 ofs=35181 (4981) nb_read: handle 4329 was not open size=4096 ofs=39277 (4982) nb_read: handle 4329 was not open size=4096 ofs=43373 (4983) nb_read: handle 4329 was not open size=4096 ofs=47469 (4986) nb_read: handle 4329 was not open size=1024 ofs=2157 (4987) nb_read: handle 4329 was not open size=4688 ofs=1667 (4988) nb_read: handle 4329 was not open size=3516 ofs=6355 (4989) nb_read: handle 4329 was not open size=16384 ofs=9871 (4990) nb_read: handle 4329 was not open size=3771 ofs=26255 (4991) nb_read: handle 4329 was not open size=2048 ofs=30026 (4992) nb_read: handle 4329 was not open size=1024 ofs=32074 (4993) nb_read: handle 4329 was not open size=6652 ofs=33098 (4994) nb_read: handle 4329 was not open size=16384 ofs=39750 (4995) nb_read: handle 4329 was not open size=10496 ofs=56134 (4996) nb_read: handle 4329 was not open size=1784 ofs=66630 (4997) nb_read: handle 4329 was not open size=90 ofs=68414 (4998) nb_read: handle 4329 was not open size=273 ofs=68504 (4999) nb_read: handle 4329 was not open size=1600 ofs=68777 (5000) nb_read: handle 4329 was not open size=640 ofs=70377 (5001) nb_read: handle 4329 was not open size=1280 ofs=71017 (5002) nb_read: handle 4329 was not open size=8464 ofs=72297 (5003) nb_read: handle 4329 was not open size=158 ofs=80761 (5004) nb_read: handle 4329 was not open size=912 ofs=80919 (5005) nb_read: handle 4329 was not open size=1980 ofs=81831 (5006) nb_read: handle 4329 was not open size=2256 ofs=83811 (5007) nb_read: handle 4329 was not open size=400 ofs=86067 (5008) nb_read: handle 4329 was not open size=252 ofs=86467 (5009) nb_read: handle 4329 was not open size=42 ofs=86719 (5011) nb_read: handle 4329 was not open size=725 ofs=86761 (5012) nb_read: handle 4329 was not open size=512 ofs=2009 (5013) nb_read: handle 4329 was not open size=9 ofs=3895 (5015) nb_read: handle 4329 was not open size=46 ofs=3781 (5017) nb_read: handle 4329 was not open size=92 ofs=4059 (5018) nb_read: handle 4329 was not open size=92 ofs=3781 (5019) nb_read: handle 4329 was not open size=184 ofs=3986 (5020) nb_read: handle 4329 was not open size=16384 ofs=3987 (5021) nb_read: handle 4329 was not open size=16384 ofs=3902 (5022) nb_read: handle 4329 was not open size=10240 ofs=3781 (5023) nb_read: handle 4329 was not open size=16384 ofs=3821 (5024) nb_read: handle 4329 was not open size=935 ofs=3891 (5025) nb_read: handle 4329 was not open size=10240 ofs=4060 (5026) nb_read: handle 4329 was not open size=184 ofs=4059 (5027) nb_read: handle 4329 was not open size=184 ofs=256 (5028) nb_read: handle 4329 was not open size=1026 ofs=256 (5029) nb_read: handle 4329 was not open size=326 ofs=3901 (5030) nb_read: handle 4329 was not open size=652 ofs=3899 (5031) nb_read: handle 4329 was not open size=2 ofs=256 (5033) nb_read: handle 4329 was not open size=4948 ofs=258 (5034) nb_close: handle 4329 was not open ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+...+......................................................................+..................................+.............................+.....................+++.........................................................................................................................+....................+...............+.....................................+......+......+.+...............................+.+....+.......................+.................+..................++....+.+..+..........++..+...........+.........+.........+..........................++++........+++++.............................+.++...........+..++++++************************************************** Throughput 27.8199 MB/sec (NB=34.7749 MB/sec 278.199 MBit/sec) Average latency 6.940010 ms Maximum latency 18993.291000 ms Latency > 2 sec. 0.081048 % Latency > 10 sec. 0.015581 % 50 clients started ...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+.................................................................+....................................................................................................................................................+..................+...+.............................................................................................................................................................................................................+..................+.......................................................+................++.......+............+.....+...............................+..+...............++.+.................................+............+.....++..++.......................+.........+...............++...........................+++.........+++.+...............+....+...+.....+........+......++....++++++.++************************************************** Throughput 27.0223 MB/sec (NB=33.7779 MB/sec 270.223 MBit/sec) Average latency 6.739319 ms Maximum latency 18699.530000 ms Latency > 2 sec. 0.090834 % Latency > 10 sec. 0.013650 % 50 clients started .........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+..............................................................................................................................+............................................................................+...................................................................................................................................................................+.+................................................................+.................................................++++...........................+..+....+..........++...........+....................+.....+.............................................+.............+................+...+..........+++....+....+............................+........+..+.....+....+...+........+........+++...............+++++..++++.++++************************************************** Throughput 27.2861 MB/sec (NB=34.1077 MB/sec 272.861 MBit/sec) Average latency 7.051013 ms Maximum latency 23801.490000 ms Latency > 2 sec. 0.091263 % Latency > 10 sec. 0.014722 % 50 clients started .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C [root@inif w]# umount /mnt [root@inif w]# e2fsck -f -y /dev/sdc e2fsck 1.38 (30-Jun-2005) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Block bitmap differences: -(1250434--1250435) -(1273360--1273370) Fix? yes
Free blocks count wrong for group #38 (32253, counted=32266). Fix? yes
Free blocks count wrong (1003250, counted=1003263). Fix? yes
/dev/sdc: ***** FILE SYSTEM WAS MODIFIED ***** /dev/sdc: 68842/640000 files (0.1% non-contiguous), 276737/1280000 blocks Nov 1 16:27:35 inif kernel: kjournald starting. Commit interval 5 seconds Nov 1 16:27:35 inif kernel: EXT3 FS on sdc, internal journal Nov 1 16:27:35 inif kernel: EXT3-fs: mounted filesystem with ordered data mode. Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:29:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9376352 Nov 1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9995808 Nov 1 16:30:19 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #608098 offset 0 Nov 1 16:30:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:31:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:31:05 inif kernel: end_request: I/O error, dev sdc, sector 367432 Nov 1 16:31:05 inif kernel: __ratelimit: 62 callbacks suppressed Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45929 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45930 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45931 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45932 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45933 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45934 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45935 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45936 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45937 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45938 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:31:22 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:31:59 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:32:45 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:33:31 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 9621088 Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 10145456 Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 7095792 Nov 1 16:34:17 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9530232 Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798712 Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798728 Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798736 Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 10059240 Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798720 Nov 1 16:35:03 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:35:49 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:36:31 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:36:31 inif kernel: end_request: I/O error, dev sdc, sector 10032752 Nov 1 16:36:35 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8862288 Nov 1 16:37:17 inif kernel: __ratelimit: 59 callbacks suppressed Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107786 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107787 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107788 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107789 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107790 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107791 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107792 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107793 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107794 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107795 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8892560 Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8893456 Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8894480 Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8895512 Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8896536 Nov 1 16:37:21 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:37:29 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9213944 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280464 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280472 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280480 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9184784 Nov 1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560038 offset 0 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10159136 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 303616 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9586656 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9641352 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10061776 Nov 1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #576172 offset 0 Nov 1 16:38:07 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:38:53 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:39:39 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:40:25 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:41:11 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:41:53 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:41:53 inif kernel: end_request: I/O error, dev sdc, sector 647544 Nov 1 16:41:53 inif kernel: __ratelimit: 639 callbacks suppressed Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80943 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80944 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80945 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80946 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80947 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80948 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80949 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80950 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80951 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80952 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:57 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:42:05 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:42:43 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:43:25 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:43:25 inif kernel: end_request: I/O error, dev sdc, sector 478480 Nov 1 16:43:25 inif kernel: __ratelimit: 118 callbacks suppressed Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59810 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59811 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59812 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59813 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59814 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59815 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59816 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59817 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59818 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59819 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:29 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:44:15 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:44:57 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:44:57 inif kernel: end_request: I/O error, dev sdc, sector 9353840 Nov 1 16:44:57 inif kernel: __ratelimit: 44 callbacks suppressed Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169230 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169231 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169232 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169233 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169234 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169235 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169236 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169237 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169238 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169239 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:45:01 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:45:11 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:45:47 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9226672 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9426536 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9832040 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9895976 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9896048 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9946768 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 279576 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 1048576 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 7097712 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9102152 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10190752 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10199216 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 295416 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9057504 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115608 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9289416 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9529432 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9577376 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9672304 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9763984 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9789792 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9811504 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9826536 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115656 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9624224 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9928144 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10003472 Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=517632, block=1250434 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10186880 Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=560474, block=1273360 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 332944 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 451768 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9208392 Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560267 offset 0 Nov 1 16:46:33 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 429128 Nov 1 16:47:15 inif kernel: __ratelimit: 67 callbacks suppressed Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 53641 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 10138096 Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267262 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267263 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267264 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267265 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267266 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267267 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267268 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267269 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267270 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 964064 Nov 1 16:47:19 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:47:28 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:47:33 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:48:05 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10082504 Nov 1 16:48:47 inif kernel: __ratelimit: 164 callbacks suppressed Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260313 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260314 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260315 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260316 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260317 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260318 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260319 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260320 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260321 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260322 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10086400 Nov 1 16:48:51 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:49:00 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:49:33 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:49:33 inif kernel: end_request: I/O error, dev sdc, sector 9777376 Nov 1 16:49:33 inif kernel: __ratelimit: 132 callbacks suppressed Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222172 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222173 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222174 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222175 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222176 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222177 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222178 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222179 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222180 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222181 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:49:46 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9728192 Nov 1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9350872 Nov 1 16:50:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709760 Nov 1 16:51:05 inif kernel: __ratelimit: 54 callbacks suppressed Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213720 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213721 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213722 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213723 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213724 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213725 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213726 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213727 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213728 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213729 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709864 Nov 1 16:51:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
| |