lkml.org 
[lkml]   [2013]   [May]   [13]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: high-speed disk I/O is CPU-bound?
On 05/10/13 20:19, Eric Wong wrote:
> Cc-ing Jens
>
> David Oostdyk <daveo@ll.mit.edu> wrote:
>> Hello,
>>
>> I have a few relatively high-end systems with hardware RAIDs which
>> are being used for recording systems, and I'm trying to get a better
>> understanding of contiguous write performance.
>>
>> The hardware that I've tested with includes two high-end Intel
>> E5-2600 and E5-4600 (~3GHz) series systems, as well as a slightly
>> older Xeon 5600 system. The JBODs include a 45x3.5" JBOD, a 28x3.5"
>> JBOD (with either 7200RPM or 10kRPM SAS drives), and a 24x2.5" JBOD
>> with 10kRPM drives. I've tried LSI controllers (9285-8e, 9266-8i,
>> as well as the integrated Intel LSI controllers) as well as Adaptec
>> Series 7 RAID controllers (72405 and 71685).
> Which I/O scheduler are you using? noop (or deadline) may improve
> things with hardware RAID.

I was using cfq, but I gave noop and deadline a try and don't see any
significant difference in my testing. Thanks for the suggestion! I had
not thought to test this yet.


>> Normally I'll setup the RAIDs as RAID60 and format them as XFS, but
>> the exact RAID level, filesystem type, and even RAID hardware don't
>> seem to matter very much from my observations (but I'm willing to
>> try any suggestions). As a basic benchmark, I have an application
>> that simply writes the same buffer (say, 128MB) to disk repeatedly.
>> Alternatively you could use the "dd" utility. (For these
>> benchmarks, I set /proc/sys/vm/dirty_bytes to 512M or lower, since
>> these systems have a lot of RAM.)
>>
>> The basic observations are:
>>
>> 1. "single-threaded" writes, either a file on the mounted
>> filesystem or with a "dd" to the raw RAID device, seem to be limited
>> to 1200-1400MB/sec. These numbers vary slightly based on whether
>> TurboBoost is affecting the writing process or not. "top" will show
>> this process running at 100% CPU.
>>
>> 2. With two benchmarks running on the same device, I see aggregate
>> write speeds of up to ~2.4GB/sec, which is closer to what I'd expect
>> the drives of being able to deliver. This can either be with two
>> applications writing to separate files on the same mounted file
>> system, or two separate "dd" applications writing to distinct
>> locations on the raw device. (Increasing the number of writers
>> beyond two does not seem to increase aggregate performance; "top"
>> will show both processes running at perhaps 80% CPU).
>>
>> 3. I haven't been able to find any tricks (lio_listio, multiple
>> threads writing to distinct file offsets, etc) that seem to deliver
>> higher write speeds when writing to a single file. (This might be
>> xfs-specific, though)
>>
>> 4. Cheap tricks like making a software RAID0 of two hardware RAID
>> devices does not deliver any improved performance for
>> single-threaded writes. (Have not thoroughly tested this
>> configuration fully with multiple writers, though.)
>>
>> 5. Similar hardware on Windows seems to be able to deliver >3GB/sec
>> write speeds on a single-threaded writes, and the trick of making a
>> software RAID0 of two hardware RAIDs does deliver increased write
>> speeds. (I only point this out to say that I think the hardware is
>> not necessarily the bottleneck.)
>>
>> The question is, is it possible that high-speed I/O to these
>> hardware RAIDs could actually be CPU-bound above ~1400MB/sec?
>>
>> It seems to be the only explanation of the benchmarks that I've been
>> seeing, but I don't know where to start looking to really determine
>> the bottleneck. I'm certainly open to suggestions to running
>> different configurations or benchmarks.
>>
>> Thanks for any help/advice!
>> Dave O.



\
 
 \ /
  Last update: 2013-05-13 17:41    [W:0.079 / U:0.984 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site