efficient network transfer

[uzon]

hi,
i'm writing a proxy for file transfers.
for example, one client sends a file to the proxy which forwards it to
another client.
i'm writing code for both the clients (send / receive file) and the
proxy (forward the file).
currently, i have this function to send the file-

InputStream fis = new FileInputStream(f.getPath() + f.separator +
param.filename);
BufferedOutputStream bos = new BufferedOutputStream(os);
do {
i = fis.read();
if (i != -1) {
bos.write(i);
}
} while (i != -1);

the forwarding code is very similar. just the input stream is a socket
not a file.
the receiving code is also similar. the output stream is to a file not
a socket.
this is working ok but it's very slow. it takes several minutes for a
file that's only a few mb.
there are two problems here i don't know how to code better-
1. loading and writing the file byte by byte is REALLY slow. how would
i go about doing it with some kind of buffer saving every X amount
received and what is the recommended amount? i want to support large
file transfers also, so saving the whole file in memory and writing to
disk in the end is not good.
2. sending byte by byte on the network (especially for the forwarder-
proxy, is also very slow). so here i also need some kind of buffer to
accept X amount and then send it.
i know there are other sending/receiving functions in java but don't
know which to use, or exactly how to use them.

any help would be greatly appreciated.
thanks in advance,
-Aaron

 

[Andrew Thompson]

hi,

Hi. Please use Upper Case letters at the start of every sentence,
Also every time you use the word 'I'. It makes posts easier to
read and understand. That is a good thing.

currently, i have this function to send the file-

This is not a 'function' (more commonly called a 'method' in Java)
so much as a 'code snippet'. It is not very helpful for understanding
what you are doing. For example..

InputStream fis = new FileInputStream(f.getPath() + f.separator +
param.filename);
BufferedOutputStream bos = new BufferedOutputStream(os);
do {
i = fis.read();

What is 'i'? Where is it defined? But no, don't
tell me, I have a better idea[1].

Usually when a programmer says 'I am having a problem with
this piece of code' thay are wrong. Because it is actually
another bit of code somewhere else that is the problem.

That is why I suggest you post a short, compilable example
of *exactly* what you are doing, from start to end (or
better, put it up at geocities or such and post a link)
for more tips on examples, check here..

i know there are other sending/receiving functions in java but don't
know which to use, or exactly how to use them.

One step is to go through the Sun IO tutorial
<http://java.sun.com/docs/books/tutorial/essential/io/>

HTH

 

[Collin VanDyck]

Usually when a programmer says 'I am having a problem with

this piece of code' thay are wrong. Because it is actually

Hello! When you are retorting wisely, please take care to double check
the spelling on your words!

Such obvious misspellings as "thay" detract from the overall readability
of your post and smack of carelessness!

For future reference, please reference

http://www.dictionary.com.

 

[Collin VanDyck]

hi,
i'm writing a proxy for file transfers.
for example, one client sends a file to the proxy which forwards it to
another client.
i'm writing code for both the clients (send / receive file) and the
proxy (forward the file).
currently, i have this function to send the file-

InputStream fis = new FileInputStream(f.getPath() + f.separator +
param.filename);
BufferedOutputStream bos = new BufferedOutputStream(os);
do {
i = fis.read();
if (i != -1) {
bos.write(i);
}
} while (i != -1);

Have you tried using a BufferedInputStream created on your InputStream?

 

[Steve Horsley]

InputStream fis = new FileInputStream(f.getPath() + f.separator +
param.filename);
BufferedOutputStream bos = new BufferedOutputStream(os);
do {
i = fis.read();
if (i != -1) {
bos.write(i);
}
} while (i != -1);

Firstly, there is a logic error here - you send the final -1 EOF
value as the last byte of the file - you shouldn't.

fis.read() goes all the way down the I/O stack making an OS call for
every byte you read. Better to wrap the FileInputStream in a
BufferedInputStream. This will make a big difference.

Maybe slightly better still would be to use a byte[] in your read
and write calls so that you transfer perhaps 1K at a time.

Steve

 

[Esmond Pitt]

(a) use BufferedInput/OutputStreams with buffers of at least 8k, or read
into and write from a byte[8192] array, not a single byte
(c) set your socket's output buffer to at least 48k
(d) if you can control the reading socket, set its input buffer to at
least 48k.

 

[Andrew Thompson]

For future reference, please reference

http://www.dictionary.com.

I'm a bit busy at the moment, why don't you do it for me?

After you check my posts[1], have a look over the 100,000 or
so pages on my web sites[2] and see if you spot any errors
there as well.

[1]
<http://groups.google.com.au/groups?as_ugroup=comp.lang.java.*&as_uauthors=andrew thompson>

[2] See sig. below.

TIA

 

[Knute Johnson]

(a) use BufferedInput/OutputStreams with buffers of at least 8k, or read
into and write from a byte[8192] array, not a single byte
(c) set your socket's output buffer to at least 48k
(d) if you can control the reading socket, set its input buffer to at
least 48k.

I'm curious how you would set your socket's buffer to 48k. Could you
explain that and what it just what it would do for you?

Thanks,

 

[Esmond Pitt]

I'm curious how you would set your socket's buffer to 48k. Could you
explain that and what it just what it would do for you?

At least 48k. on xDSL it should be nearly 64k. The buffer size should at
least equal the bandwidth-delay product. Otherwise you don't have the
maximum amount of data in flight so you aren't using the maximum
available network capacity.

EJP

 

[Knute Johnson]

At least 48k. on xDSL it should be nearly 64k. The buffer size should at
least equal the bandwidth-delay product. Otherwise you don't have the
maximum amount of data in flight so you aren't using the maximum
available network capacity.

EJP

I'm still not clear on how you would set the buffer size of a socket in
Java. I think the socket buffer size is a function of the TCP/IP stack.
I'm not a TCP/IP expert but if you know how to adjust it on my
Winbloze XP machine I would like to know.

Thanks,

 

[Esmond Pitt]

I'm still not clear on how you would set the buffer size of a socket in
Java.

java.net.Socket.setSendBufferSize()
java.net.Socket.setReceiveBufferSize()

http://java.sun.com/j2se/1.5.0/docs/api/java/net/Socket.html#setSendBufferSize(int)
http://java.sun.com/j2se/1.5.0/docs/api/java/net/Socket.html#setReceiveBufferSize(int)

 

[uzon]

andrew, this is just a forum. don't be so dramatic. troll.

the rest, thanks for responding.
you guys were right, using the buffers made a huge difference.
the transfer is more than three times faster now.
-Aaron

 

[Andrew Thompson]

andrew, this is just a forum.

Aaron/uzon, they were just suggestions (and I am fully
aware of what this is).

..don't be so dramatic. ..

Don't be so self-infatuated.

..troll.

Lovely. You cannot respond in any intelligent manner,
so you start name-calling instead.

 

[uzon]

keep your "suggestions" to yourself. this is a java forum not grammar and spelling.
troll isn't name calling. it's a description of your precious and unique attitude.

 

[Andrew Thompson]

keep your "suggestions" to yourself.

Plonk whoever your wish.

this is a java forum

Yes, it's a discussion forum for the Java programming
language, conducted primarily in English.

Such discussions are made simpler for all concerned (including
the large number of contributors who speak English as a second
language) if the poster makes every effort to be understood.

This effort amounts to including such boring and mundane things
as putting a space between paragraphs and capitalising the first
letter of sentences.

...not grammar and spelling.

You seem to be ignoring the 15 lines of my post that did not
mention grammar and spelling. For reference, I'll link to it..
<http://groups.google.com/[email protected]>

You might even visit, and benefit from, the links I included.

troll isn't name calling. it's a description of your precious and unique attitude.

Pot, kettle, black. Get over it, and please locate your shift key.

 

[uzon]

Plonk whoever your wish.


Yes, it's a discussion forum for the Java programming
language, conducted primarily in English.

Such discussions are made simpler for all concerned (including
the large number of contributors who speak English as a second
language) if the poster makes every effort to be understood.

This effort amounts to including such boring and mundane things
as putting a space between paragraphs and capitalising the first
letter of sentences.


You seem to be ignoring the 15 lines of my post that did not
mention grammar and spelling. For reference, I'll link to it..
<http://groups.google.com/[email protected]>

You might even visit, and benefit from, the links I included.


Pot, kettle, black. Get over it, and please locate your shift key.

ok i get it, telling people to use an upper case 'I' makes your day.
fine. you're great. wonderful. you make the forums worth reading.
<applause>
notice others didn't have a problem understanding my question.
fin

 

[Thomas Schodt]

notice others didn't have a problem understanding my question.

Maybe many others either
- did not bother reading your post because it was hard to read, or
- did not bother telling you your post was hard to read since Andrew had
beat them to it.

OFC you are free to continue to throttle the responses you get
by making your posts hard to read.

 

[Esmond Pitt]

Maybe many others either
- did not bother reading your post because it was hard to read, or
- did not bother telling you your post was hard to read since Andrew had
beat them to it.

OFC you are free to continue to throttle the responses you get
by making your posts hard to read.

I really must jump in here. The original post wasn't hard to read at
all. It only had two things wrong with it: a missing declaration for
'int i;' and an absence of capitalization. The declaration was obvious
and IMHO didn't need to be spelt out. I don't like miscapitalization
myself, but this was not the first posting with this these problems and
it won't be the last. I had no trouble understanding it whatsoever and
it's a shame that so much bandwidth has been wasted on all this
nitpicking. It is notable that none of the nitpickers had any idea of a
solution. It's also notable that the nitpicking has continued long after
the solution was provided. Get over it.

EJP

 

[Thomas Weidenfeller]

I really must jump in here. The original post wasn't hard to read at
all.

It was, at least for me as a non-native speaker. That's why I ignored
it. Well, I didn't ignore it immediately, I had a few unfriendly
thoughts and then ignored it.

But what in general gets worse is the attitude of posters like "uzon",
or someone going by the name of "Mike Crabs ([email protected])"
Message-ID: <[email protected]>,
Message-ID: <[email protected]>.

They post anonymously, via google-groups, often posting homework, and
simply can't behave at all. They don't take any advice, treat us as
there servants and demand to be spoon-feed.

This year is the worst I have seen here. There was usually a peek of
such people at the start of the US university term each year, which
quickly came and quickly went away. But this year it just doesn't stop.
And it is not just in .programmer, it happens in all the c.l.j.* groups
which I read. Is there a new cheating contest going on at US universities?

It's also notable that the nitpicking has continued long after
the solution was provided.

It was bad to provide a solution. This way you encourage the behavior of
people like "uzon" or "Mike Crabs". They just learned that all it takes
to get their problems fixed it to insult us and treat us like scum, that
they just have to push a little bit harder and unfriendlier to get an
answer.

IMHO, the harder they push, the less useful answers they should get, and
the more ridicule they should get. Either they learn and behave, or they
go away.

/Thomas

 

[uzon]

you'd be surprised how many people just don't care.