Messages in this thread | | | Date | Sat, 24 Jan 2004 10:27:30 +0100 | From | Josef Möllers <> | Subject | Kernel hang under extremely high load |
| |
Hi,
We're having problems with a SuSE Linux Enterprise Server 7 2.4.18-275 kernel under very high load (A SAP system using an Oracle database and Networker backup). This happens on several systems!
Shortly (some 1.5 hours) after the backup is started (usually at 10pm), kinoded went into 100% system mode, makeing the system unavailable to users. I thought I had found a bug in fs/buffer.c, size_buffers_type[] being 32 bits and counting bytes (the system has 8GB of main memory). I discussed this with Andrea Arcangeli and he said that it was unlikely that this counter overflowed, but offered a patch, nonethless (it changed the counting from bytes to 512 byte chunks). I have installed this patch on one of the systems and it now looks slightly more stable. I also added two lines to check wither the count has bit 23 set (which would indicate an overflow in the non-patched case), but, so far, this hasn't happened.
The system is now more stable, but yesterday it killed a few Oracle and SAP processes due to kernel oopses. As the system was still usable, I was able to trace one of the oopses to the following:
The "page" pointer in __block_write_full_page pointed at the following data
c54b37f0: c630b310 c34d0490 cbde3e74 0001a018 c54b3800: c6fbcc70 00000004 0200884d c4fbc37c c54b3810: c37bab7c c3a6a5c0 d32782a0 00000000
which does resemble a valid struct page (kernel virtual addresses where they should be, more numeric looking values where indexes or counts are, reasonably looking "flags"). The value at offset 0x28 (d32782a0) is supposed to be struct buffer_head * buffers; It points to the following data
d32782a0: 34323534 0287b08b 20012001 012c3a0c d32782b0: 3030030c 4c500431 300c4e51 304e3130 d32782c0: 34313030 0a323738 37313030
which does not really resemble anything like a struct buffer_head. The oops happened when the kernel tried to use 0x37313030 as
struct buffer_head *b_this_page;
One other of the Oopses which killed processes had d3278240 in one of his registers (unfortunately they had to reboot the system before I had a chance to analyze the other oopses).
My currenty assumption is that the system is extremely tight on free and available memory, the backup reading in masses of data from disk (a CLARiiON box connected through a 2GB FC link) and writing it out to tape (a tape library connected through a 1GB FC link). Now some kernel component (presumably in the buffer cache or paging area) tries to allocate memory, none is available, so it gets NULL but doesn't seem to realize this, uses it and, as a consequence, mangles some pointers.
Over the last couple of weeks we have seen some 29 oopses on one machine, all of which were in __block_write_full_page+108/432 __block_write_full_page+92/432 kmem_cache_alloc_batch+50/188 kmem_cache_alloc_batch+99/188 sync_page_buffers+20/184 sync_page_buffers+36/184 this makes me think that it's a problem somewhere in mm.
Any ideas? I'm slowly running out of 'em (and time's running out on us).
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |