Aceshardware

ajensen · Joined: 01 Sep 2007 Posts: 117

What would you use to copy massive amounts of files over network between any combination of Linux and Windows hosts? I need high performance, mirroring/sync and created/modified dates should be same on source and destination.

To top it of I just read this:
http://stevenf.com/archive/dont-use-ftp.php

But I guess I have to go with some kind of FTP software anyway.

lux_interior · Joined: 26 Jul 2007 Posts: 235

rsync is the standard tool for this kind of things. Apparently there is even a Windows port.

Del · Joined: 09 Aug 2007 Posts: 99

lux_interior wrote:

rsync is the standard tool for this kind of things. Apparently there is even a Windows port.

Indeed they have, thanks for the information.

You can use rsync stand-alone between the machines, or in conjunction with ssh if you need privacy (e.g. over internet from your laptop). I support the recommendation of using rsync, unless there is something more to the question than what I see now.

ajensen · Joined: 01 Sep 2007 Posts: 117

Thanks for the tip. rsync seems fine and I'll test it after summer holly days, which starts in 15min, got to run. :) I wonder if it can maintain high bandwidths. The algorithm seems quite CPU intensive. I was hoping to get ~2Gbit/s sustained. Penryn based platforms.

lux_interior · Joined: 26 Jul 2007 Posts: 235

ajensen wrote:

Thanks for the tip. rsync seems fine and I'll test it after summer holly days, which starts in 15min, got to run. :) I wonder if it can maintain high bandwidths. The algorithm seems quite CPU intensive. I was hoping to get ~2Gbit/s sustained. Penryn based platforms.

I'd be surprised if you couldn't tweak algorithmic complexity and compression level using command-line switches.
However, 2Gbit/s sounds high unless your filesystem cache is warm or your disks are themselves very fast...

AtWork · Joined: 31 Jul 2007 Posts: 116

ajensen wrote:

I was hoping to get ~2Gbit/s sustained.

I do a lot of work with large imagery. Usually when we're talking disk to disk on the same machine, we discuss MB/s and Gb/s is reserved for talking networks. 10Gbe ports are still pretty expensive. You can, however, use a crossover between a pair of NXB-10GCX4's for just over $1k.

Sustained writes will be your biggest bottleneck. Hardware Raid units claim up to 850 MB/s (~7Gbit/s) sustained, but, in our experience, even configuring the RAIDs with 1M blocks and transferring only very large files, you're lucky to get half that in real world usage. We are using RAID 5s for this sort of work and the controller is usually the bottleneck. RAID 0 would get you better speed, but it would be a drag to lose 8 to 12 terabytes of data every time any one of 16 disks failed....

:-)