simultaneous copy to multiple media

[Swaroop C H]

I would like to save time copying the same file
(>6 GByte) to various different target storage
media connected to the system via USB.

Is there a (Python or other) tool able to help me
to do this, so that I don't need to copy the
source file first to the first media, then to the
second, etc.?

If you are using *nix, maybe you can use the `tee` command[1] and
redirect the file to different places.

For example,

cat <somefile> | tee file1 | tee file2 > file3

I haven't tried it out but it should work.

[1]: http://unixhelp.ed.ac.uk/CGI/man-cgi?tee

HTH,

 

[Jacek Trzmiel]

I would like to save time copying the same file
(>6 GByte) to various different target storage
media connected to the system via USB.

Is there a (Python or other) tool able to help me
to do this, so that I don't need to copy the
source file first to the first media, then to the
second, etc.?

Use 'tee' command from shell:
tee <file >/dest1/file /dest2/file

Or run two copies in parallel (one of them probably will read file from
cache):
cp file /dest1/file & cp file /dest2/file


-- Jacek.

 

[Claudio Grondi]

I would like to save time copying the same file
(>6 GByte) to various different target storage
media connected to the system via USB.

Is there a (Python or other) tool able to help me
to do this, so that I don't need to copy the
source file first to the first media, then to the
second, etc.?

Claudio

 

[Claudio Grondi]

I am on a Widows 2000 box using the NTFS file system.
Both up to now suggested approaches as
- tee.exe (where I used the http://david.tribble.com/dos/tee.exe
DOS-port with redirecting stdout to NULL) and
- parallel copy (hoping caching does the job) are by far
slower than the consecutive copy:
single copy speed 12-15 MByte/s
which gives effectively
6-7 MByte/s,
tee.exe or twice copy in parallel
1-3 MByte/s.

Any other suggestions except writing an own
optimised version of tee.exe, because it is so
easy to come up with:
while(c=getchar()){fputc(c,fp1);fputc(c,fp2);}
?

Is there maybe a way to use a direct DMA
transfer to multiple target destinations
(I copy to drives connected via USB ports) ?

Can someone point me in the right direction?

Claudio

 

[Grant Edwards]

Is there maybe a way to use a direct DMA
transfer to multiple target destinations
(I copy to drives connected via USB ports) ?

Sure, but you'll have to throw away your OS, and write a
program that runs on the bare metal.

Can someone point me in the right direction?

I think they already did.

 

[Claudio Grondi]

means your message, that you think, that
the consecutive copy is the fastest possible
method if using Windows 2000?

Claudio

 

[Rolf Zwart]

[...]
If you are using *nix, maybe you can use the `tee` command[1] and
redirect the file to different places.

For example,

cat <somefile> | tee file1 | tee file2 > file3

On Unixes I know, only 1 process is needed:
<infile tee outfile1 outfile2 outfile3 ... >/dev/null
It does work!

** Rolf

I haven't tried it out but it should work.

[1]: http://unixhelp.ed.ac.uk/CGI/man-cgi?tee

HTH,

 

[Heiko Wundram]

Is there maybe a way to use a direct DMA
transfer to multiple target destinations
(I copy to drives connected via USB ports) ?

Think about what USB stands for. Then reconsider whether you'll ever have the
chance of writing truly simultaneously to several devices connected via
USB... And then, as an extra exercise, think about why it takes so long when
several different jobs are done in parallel, writing to devices connected via
USB.

Bus design... Don't they teach anything at uni these days?

--
--- Heiko.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQBCPeeqf0bpgh6uVAMRAopNAJ9fLMRkc3rCt6kfowV4Nyn6gKvI2wCeNdvd
giHebgCQT+7HPGDn6So99tU=
=UiN+
-----END PGP SIGNATURE-----

 

[Diez B. Roggisch]

2) if I understand it right, an USB controller is connected to

to the PCI bus and there can be many separate USB
controller on one PC
True.

3) the theoreticall speed of USB (430 MByte/s?) is much
higher as the experienced 15 MByte/s, probably due to slow
controller on the side of the external storage media, so
maybe even on one USB line I will have the chance to
use it full capacity connecting many slow devices
(I can't imagine, that USB goes down with the speed to
the speed of the slowest component connected, does it?)

Nope. Or yes. Here comes into play why Heiko said think of what USB stands
for. While the devices appear to be responsive concurrently (the bus
arbitration and multiplexing/demultiplexing is tranparent to the user), it
still is a serial bus, so at a given point in time you can only write to
_one_ device. And if that device is limited in its speed, you're doomed.
The only thing I could imagine is that if you send small bursts that don't
make the write cache full, it _might_ be possible to switch fast between
the devices and thus write the data faster, as the devices can try and
write the data to disk while the others are feeded.

But I'm on no sure ground here with that assumption, and it certainly would
depend on how the usb-ata-controller deals with that. You said yourself
that you only had 15MB/sec, so it doesn't look to good.

To make this work would certainly be a deep lowlever driver hack and nowhere
in the scope of python per se.

4) with 45 MByte/s I could theoretically simultaneously
supply three separate data streams to three USB
controller writing with 15 MByte/s each out to the
external storage media.

That might be the case if you use three real separate controllers. But that
will depend on how the OS deals with these, if you really achieve the
theoretical throughoutput.

5) watching the low CPU usage while writing to USB
shows me, that CPU-time is not a problem.

That is certainly true - the bottleneck are the buses.

 

[Diez B. Roggisch]

Nope. Or yes. Here comes into play why Heiko said think of what USB stands
for. While the devices appear to be responsive concurrently (the bus
arbitration and multiplexing/demultiplexing is tranparent to the user), it
still is a serial bus, so at a given point in time you can only write to
_one_ device. And if that device is limited in its speed, you're doomed.
The only thing I could imagine is that if you send small bursts that don't
make the write cache full, it _might_ be possible to switch fast between
the devices and thus write the data faster, as the devices can try and
write the data to disk while the others are feeded.

I had second thoughts of this, and have to admit that I don't make too much
senes here. So ignore that.

 

[Claudio Grondi]

I don't know any deep details about USB, except that
I _know_ that it is a serial bus, but considering
following:

1) I can read/write 45 MByte/s from harddrive to harddrive
on the E-IDE bus (theoretically 100 MByte/s), so the
speed of the harddrive I read/write from/to is probably the
bottleneck.
2) if I understand it right, an USB controller is connected to
to the PCI bus and there can be many separate USB
controller on one PC
3) the theoreticall speed of USB (430 MByte/s?) is much
higher as the experienced 15 MByte/s, probably due to slow
controller on the side of the external storage media, so
maybe even on one USB line I will have the chance to
use it full capacity connecting many slow devices
(I can't imagine, that USB goes down with the speed to
the speed of the slowest component connected, does it?)
4) with 45 MByte/s I could theoretically simultaneously
supply three separate data streams to three USB
controller writing with 15 MByte/s each out to the
external storage media.
5) watching the low CPU usage while writing to USB
shows me, that CPU-time is not a problem.

I still see a chance of writing truly simultaneously to
several devices connected via USB.
Please let me know if I am wrong in any of the
points (1-5) or if my conclusion is based on
wrong assumptions or if I use a wrong way of
infering.

By the way: I was not thought about bus design at
university, so what I know about it is comes from
self-study .

Claudio
P.S. Your message appears in my Outlook Express
as one having no content, but an attachment, so I can't
read it directly.

Is there maybe a way to use a direct DMA
transfer to multiple target destinations
(I copy to drives connected via USB ports) ?

Think about what USB stands for. Then reconsider whether you'll ever have
the
chance of writing truly simultaneously to several devices connected via
USB... And then, as an extra exercise, think about why it takes so long when
several different jobs are done in parallel, writing to devices connected
via
USB.
Bus design... Don't they teach anything at uni these days?

 

[Jacek Trzmiel]

I am on a Widows 2000 box using the NTFS file system.
Both up to now suggested approaches as
- tee.exe (where I used the http://david.tribble.com/dos/tee.exe
DOS-port with redirecting stdout to NULL) and
- parallel copy (hoping caching does the job) are by far
slower than the consecutive copy:
single copy speed 12-15 MByte/s
which gives effectively
6-7 MByte/s,
tee.exe or twice copy in parallel
1-3 MByte/s.

Any other suggestions except writing an own
optimised version of tee.exe

Try this:
http://mastermind.com.pl/multicopy/

This is small tool I've wrote, that does use large memory buffers with
asynchronous I/O to copy file.

Following command:
multicopy c:\testfile d:\testfile e:\testfile f:\testfile
will copy c:\testfile to d, e and f disks.

With four separate IDE disks I can copy file at about 30MB/s, which
means 120MB/s total I/O. You can give it a try, but I don't know if it
will work fast with USB drives.

HTH,
sc0rp.

 

[Heiko Wundram]

Am Sonntag, 20. März 2005 23:16 schrieb Claudio Grondi:

2) if I understand it right, an USB controller is connected to
to the PCI bus and there can be many separate USB
controller on one PC

Yes, there may be more than one USB-controller in a PC, but this doesn't
matter, they are all connected to the same bus (the PCI bus). This is a
parallel bus, but also a bus which can only serve a single endpoint at a
time. The frequency is normally 66 Mhz, so you get a theoretical 66*32 Mb/s
over the PCI bus.

But, this is only theoretical usage. Consider the EIDE-controller trying to
write data into DMA memory which it has read from disk. It has to do the
following to communicate with memory:

1) signal the DMA chip (which is also a device on the PCI bus) that it has
data. This implies grabbing the PCI bus, making sure noone else is talking at
the moment.

2) pass the data, 32 bit at a time, to the DMA chip, which places this data
into main memory.

3) you may not grab the bus too long, as other devices might also want to
communicate with some other device (e.g. the PIC, to signal an interrupt to
the CPU). Thus, you have to free the bus after some time, and you have to
forcibly give free the bus when some other device signals an interrupt. This
also means starting transmission again after the bus has been successfully
reacquired.

4) finally, after communication with the DMA-chip is done, signal the PIC via
an interrupt that you wish the CPU to be signaled that it should do a
software interrupt, so that the OS interrupt handler can be scheduled and can
take the data from the DMA region (and do something with it).

Now, even when the bus is fast, setting it up all still takes time, so you
never get the theoretical throughput you might wish to see.

Accessing the USB controller does the game the other way around; the OS uses
the DMA-chip to stream the data onto the PCI bus for the controller to
transmit.

So, overall, adding more USB controllers to your PC only adds extra throughput
until you reach the top limit of PCI (when the controllers start stealing
bandwith from each other).

3) the theoreticall speed of USB (430 MByte/s?) is much
higher as the experienced 15 MByte/s, probably due to slow
controller on the side of the external storage media, so
maybe even on one USB line I will have the chance to
use it full capacity connecting many slow devices
(I can't imagine, that USB goes down with the speed to
the speed of the slowest component connected, does it?)

Well, 430 MB/s is only for USB 2.0. AFAIK, most devices (esp. storage devices)
are still only USB 1.1 compliant, which makes this rate go down to a mere 40
MB/s or something close.

4) with 45 MByte/s I could theoretically simultaneously
supply three separate data streams to three USB
controller writing with 15 MByte/s each out to the
external storage media.

No, you can't. As I said in response to point 2, it doesn't matter what kind
of bus you use, when the bus isn't meant to do real "broadcasts" or can
actually talk to several devices simultaneously because all wires are
duplicated a corresponding number of times, there's always overhead when
switching between the devices.

The bus must be acquired for communication with one device (so that the others
won't simply go blabbering over the communication), then the data must be
streamed, then the bus released again. This process takes time, and AFAIK,
USB is pretty expensive (compared to data transmission speed) here, as USB
guarantees that a packet reaches the endpoint when its transmitted over the
wire (in the sense that it won't get lost because someone else was also
talking). Now, when more devices come into play, this overhead becomes
significant, and takes away quite some bandwith.

For example, Ethernet takes another way. Ethernet simply says: when I have
data which is transmitted while someone else was talking, throw away the
resulting mess (a collision, remember those cool blinking lights on hubs?).
Someone will retransmit it if it was important.

5) watching the low CPU usage while writing to USB
shows me, that CPU-time is not a problem.

No, it most certainly isn't. That's why modern computers always use a DMA-chip
to do the heavy work of dealing with the hardware on external busses...

P.S. Your message appears in my Outlook Express
as one having no content, but an attachment, so I can't
read it directly.

Err, that's pretty strange. I can read my KMail written emails in pretty much
any client that I have access to... So I guess this is a problem with Outlook
Express not understanding MIME correctly, and not a problem on my side...
And, don't worry, I won't switch to <windows sucks>-mode now.

--
--- Heiko.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQBCPgR0f0bpgh6uVAMRAvmQAJ0UOF+foOuTktKgXT4C1Ghhz7loEACfTtQb
5qASZyT1uJfoMw4qjS7/pSU=
=0Yv+
-----END PGP SIGNATURE-----

 

[Diez B. Roggisch]

Well, 430 MB/s is only for USB 2.0. AFAIK, most devices (esp. storage

devices) are still only USB 1.1 compliant, which makes this rate go down
to a mere 40 MB/s or something close.

I think it is 430 MBit(!), so about 50MB per second. The old usb 1.1 has
12MBits. So I don't think that Claudio could observe 15MByte/s with a 1.1
bus....

No, you can't. As I said in response to point 2, it doesn't matter what
kind of bus you use, when the bus isn't meant to do real "broadcasts" or
can actually talk to several devices simultaneously because all wires are
duplicated a corresponding number of times, there's always overhead when
switching between the devices.

Well, if the bus is faster, it can (and does) use multiplexing. All busses
do that, apart from crossbar switches which have n:n point2point
connectivity.

The bus must be acquired for communication with one device (so that the
others won't simply go blabbering over the communication), then the data
must be streamed, then the bus released again. This process takes time,
and AFAIK, USB is pretty expensive (compared to data transmission speed)
here, as USB guarantees that a packet reaches the endpoint when its
transmitted over the wire (in the sense that it won't get lost because
someone else was also talking). Now, when more devices come into play,
this overhead becomes significant, and takes away quite some bandwith.

USB uses a host-based bus arbitration scheme where only the controller
decides who can talk when. So there is very limited bus arbitration
overhead. But of course this also means that the host has grant each
attached device bus time every now and then, as the device itself has no
chance of signalling data availability itself. So - that adds again some
overhead

In the end, I'm not sure what throughoutput is actually possible. As I said
to Claudio in a private mail: Apart from multithreaded writing to the bus
and _hoping_ that things speed up one can't do much - at least not in
python, and not without deep driver fiddling or even writing drivers
oneself.

 

[Dennis Lee Bieber]

3) the theoreticall speed of USB (430 MByte/s?) is much

USB 2.0... ~440 Mbit/sec

USB 1.1 is around 11Mbps

USB 1.0 <2Mbps

FireWire... 400Mbps

10Base-T Ethernet, 10Mbps (USB 1.1 just beats it in theory)
100Base-T Ethernet (aka EtherFast) 100Mbps
Gigabit Ethernet 1000Mbps

USB/Ethernet/Firewire are serial protocols, Your IDE (or SCSI)
drives are parallel

(I can't imagine, that USB goes down with the speed to
the speed of the slowest component connected, does it?)

Yes, it does... Plug a 1.1 device into a 2.0 bus, and the bus
controller down-shift the whole thing to 1.1 speeds.

But if you mean data rate /within/ the device... You'll get a
polling situation. Data will transfer at the USB rate, but you may have
pauses while the device fills/empty internal buffers.

I still see a chance of writing truly simultaneously to
several devices connected via USB.

The protocol is still serial... You may have submitted three
write commands, to three different devices, but the USB driver has to
format the data into packets for each device, sending one after the
other... Most likely, for such a drive, 512/1K bytes (4/8Kbits plus
overhead). If you are lucky, the first drive will still be processing
the first packet by the time the driver comes around to sending it its
second packet.


--

 

[Claudio Grondi]

Try this:

http://mastermind.com.pl/multicopy/

This is small tool I've wrote, that does use large memory buffers with
asynchronous I/O to copy file.

Thank you!
This (with a drawback of blocking the entire system) does it!
( dzieñ dobry i dziêkujê za t± konstruktywn± odpowied¼
na moje pytanie )

At least I know, that it works and you explain how it works,
so I am now quite sure, that it will be also possible to have
a Python tool which does the same thing.
This way the discussion going towards driver hacking for
the USB access was in my case just the wrong way, but
anyway, thanks also for that to the contributors, because it
sharpened my understanding of USB.

Here my current test results on my Windows 2000 system
with a 2.8 GHz P4 CPU and two 5 1/4" IDE-USB 2.0 casings
(I have got from http://www.computeruniverse.net/ a German
online shop, article no.: 90052509)
by the way:
does someone know an IDE-USB case able
to perform data transfer faster than the
max. 20 MByte/s I experience with one of my
another IDE-USB 5 1/4" casings?
?
) :
Single file copy:
USB-1 to USB-2 = 11 MB/s, max. 70% CPU
USB-2 to USB-1 = 11 MB/s, max. 70% CPU
E-IDE to USB-1 = 12 MB/s, max. 70% CPU (first time)
E-IDE to USB-1 = 13 MB/s, max. 70% CPU (second time same file)
E-IDE to E-IDE = max. 45 MB/s (two physical drives)

Simultaneous file copy:
USB-1 to USB-2 = 5 MB/s
USB-2 to USB-1 = 7 MB/s
-------
12 MB/s, max. 80% CPU

Simultaneous file copy using
http://mastermind.com.pl/multicopy/release/1.0.0/multicopy.exe :
E-IDE to USB-1 = 10 MB/s
E-IDE to USB-2 = 10 MB/s
-------
20 MB/s, unknown CPU load (1)

(1) work with the PC during copying not possible,
system "hangs" from 5 to 15 seconds between
e.g. displaying current system time (with
Windows clock).

From my point of view this thread has reached
its end (I have a solution I can live with), except if
someone would like to contribute or point to a
better multicopy.exe which does not block the system
or best to contribute or point to a _Python_ script or
module which is able to find out the optimal buffer
size for copying on the
current system (and the best way of copying files),
so that after this information is saved in an .INI file
the tool is best adopted to the system it works on.

I am quite sure, that there is a perfect ready-to-use
solution out there (up to now I just only failed to find
one by Googling, e.g. the tee.exe provided by
http://unxutils.sourceforge.net/ doesn't work with
large files at all), so please don't hesitate to provide
it, so, that I don't need to "reinvent the wheel".

Claudio

 

[Jacek Trzmiel]

Thank you!
This (with a drawback of blocking the entire system) does it!
( dzieñ dobry i dziêkujê za t± konstruktywn± odpowied¼
na moje pytanie )


From my point of view this thread has reached
its end (I have a solution I can live with), except if
someone would like to contribute or point to a
better multicopy.exe which does not block the system

Symptoms (high cpu usage, unresponsive system) look similar to situation
when you try to read/write as fast as possible from/to IDE drive running
in PIO mode. I think that it's either USB driver problem, or inherent
design flaw in USB (anyone?).

Anyway, I've added buffersize and sleeptime options to multicopy, so you
may try to throttle it down. Download it here:
http://mastermind.com.pl/multicopy/

HTH,
sc0rp.

 

[Bengt Richter]

Symptoms (high cpu usage, unresponsive system) look similar to situation
when you try to read/write as fast as possible from/to IDE drive running
in PIO mode. I think that it's either USB driver problem, or inherent
design flaw in USB (anyone?).

Anyway, I've added buffersize and sleeptime options to multicopy, so you
may try to throttle it down. Download it here:
http://mastermind.com.pl/multicopy/

What if some disks could benefit from running ahead a few buffers while others
are hanging slowed by e.g. allocation and seeking activity? ISTM there could be
a benefit to keeping a multibuffer readahead window of the source stream going?
(I didn't look at your code, maybe you do this? Also maybe a particular OS might
do this for you so that using several open source streams coming from the same datafile
would automatically share system readahead buffer info if reads stayed within
a few buffers of each other. Do you us multiple open readonly files as source streams?
Or are OS file systems so dumb they don't notice shareability of temporarily
memory-resident readonly file data buffers? I guess it would vary, and you could
lose or gain by single or multifile source strategy, depending ;-)

Regards,
Bengt Richter

 

[Christos TZOTZIOY Georgiou]

On Unixes I know, only 1 process is needed:
<infile tee outfile1 outfile2 outfile3 ... >/dev/null
It does work!

Of course it does. Why wouldn't it? Or are you referring to the fact
that redirection can appear anywhere on the command line?

However, >/dev/null is a waste. Substitute one of the outfiles instead.

 

[Christos TZOTZIOY Georgiou]

Well, 430 MB/s is only for USB 2.0.

It's 480 Mb/s (megabit or mebibit, I am not sure... , so it maxes at
about 60 MB/s (or MiB/s) for all devices on the same controller.