Messages in this thread |  | | From | Denys Fedoryshchenko <> | Subject | probably XFS regression 2.6.24.x to 2.6.25-rc7-git5, XFS very unstable | Date | Sat, 5 Apr 2008 10:11:33 +0300 |
| |
On loaded proxies running under 2.6.25-rc7-git one, and 2.6.25-rc8 i have at night outages. The filesystem is crashing without any serious reason (there is no power outage or even reboot). Even i did xfs_repair, after while it is crashing again, so probably or there is my mistake or regression kind of very serious. On some proxies i did kexec while upgrading kernel, and probably filesystem was not unmounted properly. But it crashes again after xfs_repair! I have absolutely same configuration proxy with 2.6.24.x, it is rock solid.
Here is dmesg i have on crash uname Linux Proxy-Karam114 2.6.25-rc7-git5-build-0026 #10 SMP Mon Mar 31 04:41:05 EEST 2008 i686 unknown mount options /bin/mount -o noatime -L CACHE1 /cache1 /bin/mount -o noatime -L CACHE2 /cache2
Proxy-Karam114 /proc/sys/fs/xfs # grep "" * age_buffer_centisecs:1500 error_level:3 filestream_centisecs:3000 inherit_noatime:1 inherit_nodefrag:1 inherit_nodump:1 inherit_nosymlinks:0 inherit_sync:1 irix_sgid_inherit:0 irix_symlink_mode:0 panic_mask:0 restrict_chown:1 rotorstep:1 stats_clear:0 xfsbufd_centisecs:100 xfssyncd_centisecs:3000
Apr 2 09:08:52 194.146.153.114 kernel: [194090.355098] xfs_force_shutdown(sdd1,0x1) called from line 420 of file fs/xfs/xfs_rw.c. Return address = 0xf8b25dd6 Crashed, probably FS was unclean, i did xfs_repair
Here FS probably unclean Apr 4 03:03:10 194.146.153.114 kernel: [350703.385931] xfs_inotobp: xfs_imap() returned an error 22 on sdd1. Returning error. Apr 4 03:03:10 194.146.153.114 kernel: [350703.385931] xfs_iunlink_remove: xfs_inotobp() returned an error 22 on sdd1. Returning error. Apr 4 03:03:10 194.146.153.114 kernel: [350703.385931] xfs_inactive: xfs_ifree() returned an error = 22 on sdd1 Apr 4 03:03:10 194.146.153.114 kernel: [350703.385931] xfs_force_shutdown(sdd1,0x1) called from line 1737 of file fs/xfs/xfs_vnodeops.c. Return address = 0xf8b257db Apr 4 03:03:10 194.146.153.114 kernel: [379153.774514] xfs_difree: xfs_inobt_lookup_le returned() an error 5 on sdd1. Returning error. Apr 4 03:03:10 194.146.153.114 kernel: [379153.838789] xfs_iunlink_remove: xfs_itobp() returned an error 5 on sdd1. Returning error. Apr 4 03:03:10 194.146.153.114 kernel: [379153.838843] xfs_iunlink_remove: xfs_trans_read_buf() returned an error 5 on sdd1. Returning error. Apr 4 03:03:10 194.146.153.114 kernel: [379153.838865] xfs_iunlink_remove: xfs_trans_read_buf() returned an error 5 on sdd1. Returning error. Apr 4 03:03:10 194.146.153.114 kernel: [379153.867218] xfs_difree: xfs_inobt_lookup_le returned() an error 5 on sdd1. Returning error. Apr 4 03:03:10 194.146.153.114 kernel: [379153.867243] xfs_iunlink_remove: xfs_trans_read_buf() returned an error 5 on sdd1. Returning error.
First errors appears in squid, nothing came to dmesg from kernel Apr 4 03:46:38 194.146.153.28 squid[1793]: cache_dir /cache3/squid: (5) Input/output error Apr 4 03:46:38 194.146.153.28 ERROR: squid stopped? status 134 Apr 4 03:46:41 194.146.153.28 ERROR: squid started Apr 4 03:46:41 194.146.153.28 squid[2073]: cache_dir /cache3/squid: (5) Input/output error Apr 4 03:46:41 194.146.153.28 ERROR: squid stopped? status 134 Apr 4 03:46:43 194.146.153.28 ERROR: squid started
After while Apr 4 06:44:26 194.146.153.28 kernel: [285436.482594] xfs_force_shutdown(sde1,0x1) called from line 420 of file fs/xfs/xfs_rw.c. Return address = 0xf8b73dd6
-- ------ Technical Manager Virtual ISP S.A.L. Lebanon
|  |