lkml.org 
[lkml]   [2008]   [Jan]   [17]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectFast network file copy; "recvfile()" ?
I need to copy large (> 100GB) files between machines on a fast
network. Both machines have reasonably fast disk subsystems, with
read/write performance benchmarked at > 800 MB/sec. Using 10GigE cards
and the usual tweaks to tcp_rmem etc., I am getting single-stream TCP
throughput better than 600 MB/sec.

My question is how best to move the actual file. NFS writes appear to
max out at a little over 100 MB/sec on this configuration. FTP and
rcp give me around 250 MB/sec. Thus I am planning to write custom
code to send and receive the file.

For sending, I believe my best options are:

1) O_DIRECT read() + send()
2) mmap() + madvise(WILLNEED) + send()
3) fadvise(WILLNEED) + sendfile()

I am leaning towards (3), since I gather that sendfile() is supposed
to be pretty fast.

My question is what to do on the receiving end. In short, if I want
the equivalent of a "recvfile()" to go with sendfile(), what is my
best bet on Linux?

I will probably try recv() + O_DIRECT write(), but I am curious if
there are other approaches I should try.

Thanks!

- Pat


\
 
 \ /
  Last update: 2008-01-17 21:57    [W:0.057 / U:0.084 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site