Messages in this thread | | | Date | Fri, 2 Jul 2004 14:01:44 +0200 | From | Markus Schaber <> | Subject | Re: Block Device Caching |
| |
Hi, Helge,
On Fri, 02 Jul 2004 13:15:35 +0200 Helge Hafting <helge.hafting@hist.no> wrote: > > [Caching problems] > >Any Ideas? > > > Not much. "bear" may have decided to drop cache in one case but not > in the other - that is valid although strange. Take a look at what > block size the device uses. A mounted device uses the fs blocksize, a > partition(or entire disk) may be set to some different blocksize. (I > believe it might be the classical 512 byte) Caching blocks > with a size different from the page size (usually 4k) has extra > overhead and could lead to extra memory pressure and different caching > decisions. > > So I recommend retesting bear with a good blocksize, such as 4k, on > the device. That also gives somewhat better performance in general > than 512-byte blocks. There is a syscall for changing the block size > of a device. > > (Block size may also be changed by the following hack: mount a fs > like ext2 on the device, then umount it. Of course you need to use > mkfs so this can't be done on a device that contains data. > "mount" sets the device block size to the size used by the fs. > After umount, the device is free to be opened directly again, but it > retains the new blocksize.)
This sounds reasonable, and so I did some tests wr/t mounting and memory pressure:
bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 775776k used, 3177848k free, 458060k buffers Swap: 1048568k total, 0k used, 1048568k free, 31812k cached
bear:~/readcachetest# ./readcache /dev/daten/testing 1000 frist run needed 12.374819 seconds, this is 80.809263 MBytes/Sec second run needed 12.004670 seconds, this is 83.300915 MBytes/Sec third run needed 11.898035 seconds, this is 84.047492 MBytes/Sec
bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 775648k used, 3177976k free, 457904k buffers Swap: 1048568k total, 0k used, 1048568k free, 31900k cached
bear:~/readcachetest# mount /dev/daten/testing /testinmount/ -t ext2
bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 775720k used, 3177904k free, 457944k buffers Swap: 1048568k total, 0k used, 1048568k free, 31860k cached
bear:~/readcachetest# ./readcache /dev/daten/testing 1000 frist run needed 12.199237 seconds, this is 81.972340 MBytes/Sec second run needed 12.028633 seconds, this is 83.134966 MBytes/Sec third run needed 12.454502 seconds, this is 80.292251 MBytes/Sec
bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 1125432k used, 2828192k free, 808008k buffers Swap: 1048568k total, 0k used, 1048568k free, 31792k cached
bear:~/readcachetest# ./readcache /testinmount/test.dump 1000 frist run needed 11.985626 seconds, this is 83.433272 MBytes/Sec second run needed 3.682361 seconds, this is 271.564901 MBytes/Sec third run needed 3.638563 seconds, this is 274.833774 MBytes/Sec
bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 2150128k used, 1803496k free, 808736k buffers Swap: 1048568k total, 0k used, 1048568k free, 1055688k cached
bear:~/readcachetest# ./readcache /dev/daten/testing 1000 frist run needed 12.418232 seconds, this is 80.526761 MBytes/Sec second run needed 12.381597 seconds, this is 80.765026 MBytes/Sec third run needed 12.185159 seconds, this is 82.067046 MBytes/Sec
bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 2149232k used, 1804392k free, 807784k buffers Swap: 1048568k total, 0k used, 1048568k free, 1055756k cached
bear:~/readcachetest# umount /testinmount/
bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 775728k used, 3177896k free, 457872k buffers Swap: 1048568k total, 0k used, 1048568k free, 31728k cached
bear:~/readcachetest# ./readcache /dev/daten/testing 1000 frist run needed 12.171772 seconds, this is 82.157306 MBytes/Sec second run needed 11.842943 seconds, this is 84.438471 MBytes/Sec third run needed 12.167059 seconds, this is 82.189131 MBytes/Sec
bear:~/readcachetest# top | head -n 8 | tail -n 2 Mem: 3953624k total, 775656k used, 3177968k free, 457876k buffers Swap: 1048568k total, 0k used, 1048568k free, 31860k cached
So files from the mounted fs get cached, but not direct reads from the block device, although the lvm volume is mounted, and enough memory is available.
I currently suspect this to be a problem with the cciss controller, see my other mail to the list...
> You may also want to test with a smaller count. Reading directly from > pagecache should be noticeably faster than reading from the cache on > the raid controller, although either obviously is way faster than > reading from actual disks.
bear:~/readcachetest# ./readcache /dev/daten/testing 100 frist run needed 1.202330 seconds, this is 83.171841 MBytes/Sec second run needed 0.349686 seconds, this is 285.970842 MBytes/Sec third run needed 0.346644 seconds, this is 288.480401 MBytes/Sec
bear:~/readcachetest# mount /dev/daten/testing /testinmount/ -t ext2
bear:~/readcachetest# ./readcache /testinmount/test.dump 100 frist run needed 1.205797 seconds, this is 82.932699 MBytes/Sec second run needed 0.367004 seconds, this is 272.476594 MBytes/Sec third run needed 0.362783 seconds, this is 275.646874 MBytes/Sec
bear:~/readcachetest# ./readcache /testinmount/test.dump 100 frist run needed 0.363097 seconds, this is 275.408500 MBytes/Sec second run needed 0.363002 seconds, this is 275.480576 MBytes/Sec third run needed 0.362852 seconds, this is 275.594457 MBytes/Sec
bear:~/readcachetest# umount /testinmount/
bear:~/readcachetest# mount /dev/daten/testing /testinmount/ -t ext2
bear:~/readcachetest# ./readcache /testinmount/test.dump 100 frist run needed 1.189227 seconds, this is 84.088235 MBytes/Sec second run needed 0.367456 seconds, this is 272.141426 MBytes/Sec third run needed 0.363139 seconds, this is 275.376646 MBytes/Sec
Hmm, I'm somehow irritated now, as 275 MB/Sec seems rather low for reads from the page cache on a 2.8Ghz Xeon machine (even my laptop gets about 400 MB/Sec with this, and Skate even reaches 3486.385664 MBytes/Sec.)
No idea... Markus. -- markus schaber | dipl. informatiker logi-track ag | rennweg 14-16 | ch 8001 zürich phone +41-43-888 62 52 | fax +41-43-888 62 53 mailto:schabios@logi-track.com | www.logi-track.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |