lkml.org 
[lkml]   [2008]   [Nov]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectRe: Slow file transfer speeds with CFQ IO scheduler in some cases
Date
Jens Axboe <jens.axboe@oracle.com> writes:

> On Mon, Nov 10 2008, Jeff Moyer wrote:
>> Jens Axboe <jens.axboe@oracle.com> writes:
>>
>> > On Sun, Nov 09 2008, Vitaly V. Bursov wrote:
>> >> Hello,
>> >>
>> >> I'm building small server system with openvz kernel and have ran into
>> >> some IO performance problems. Reading a single file via NFS delivers
>> >> around 9 MB/s over gigabit network, but while reading, say, 2 different
>> >> or same file 2 times at the same time I get >60MB/s.
>> >>
>> >> Changing IO scheduler to deadline or anticipatory fixes problem.
>> >>
>> >> Tested kernels:
>> >> OpenVZ RHEL5 028stab059.3 (9 MB/s with HZ=100, 20MB/s with HZ=1000
>> >> fast local reads)
>> >> Vanilla 2.6.27.5 (40 MB/s with HZ=100, slow local reads)
>> >>
>> >> Vanilla performs better in worst case but I believe 40 is still low
>> >> concerning test results below.
>> >
>> > Can you check with this patch applied?
>> >
>> > http://bugzilla.kernel.org/attachment.cgi?id=18473&action=view
>>
>> Funny, I was going to ask the same question. ;) The reason Jens wants
>> you to try this patch is that nfsd may be farming off the I/O requests
>> to different threads which are then performing interleaved I/O. The
>> above patch tries to detect this and allow cooperating processes to get
>> disk time instead of waiting for the idle timeout.
>
> Precisely :-)
>
> The only reason I haven't merged it yet is because of worry of extra
> cost, but I'll throw some SSD love at it and see how it turns out.

OK, I'm not actually able to reproduce the horrible 9MB/s reported by
Vitaly. Here are the numbers I see.

Single dd performing a cold cache read of a 1GB file from an
nfs server. read_ahead_kb is 128 (the default) for all tests.
cfq-cc denotes that the cfq scheduler was patched with the close
cooperator patch. All numbers are in MB/s.

nfsd threads| 1 | 2 | 4 | 8
----------------------------------------
deadline | 65.3 | 52.2 | 46.7 | 46.1
cfq | 64.1 | 57.8 | 53.3 | 46.9
cfq-cc | 65.7 | 55.8 | 52.1 | 40.3

So, in my configuration, cfq and deadline both degrade in performance as
the number of nfsd threads is increased. The close cooperator patch
seems to hurt a bit more at 8 threads, instead of helping; I'm not sure
why that is.

Now, the workload that showed most slowdown for cfq with respect to
other I/O schedulers was using dump(8) to backup a file system. Here
are the numbers for that:

deadline 82241 kB/s
cfq 34143 kB/s
cfq-cc 82241 kB/s

And a customer actually went to the trouble to write a test to approximate
dump(8)'s I/O patterns. For that test, we also see a big speedup (as
expected):

deadline 87157 kB/s
cfq 20218 kB/s
cfq-cc 87056 kB/s

Jens, if you have any canned fio workloads that you use for regression
testing, please pass them my way and I'll give them a go on some SAN
storage.

Cheers,

Jeff


\
 
 \ /
  Last update: 2008-11-12 19:35    [W:0.171 / U:0.048 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site