problem reading TCP packets on socket

[antoine]

Hello,

I'm having a pretty annoying issue with a TCP connection in a
client/server environment.

my client connects to a server that sends it messages of variable
lengths.

on reception of each message, the client strips down the message in the
following manner:

1. the first two characters of the message are first read to indicate
the length of the message (as in the "decodeLength" method in the code
below)
2. knowing the length of the message, I read the appropriate number of
characters on my bufferedReader, and I start over.

this method has proven effective for a very long time for me, but
recently I installed a new server on a slightly different environment
(stable though), and started experiencing strange things. after
tracking down the errors, I realized that problems arised when TCP
packets were not received properly, and the TCP stack asked for a
resend.

I haven't exactly figured out what's happening at the TCP layer yet,
but I "trust" the TCP stack, and think my client should be able to
handle such situation, however it does not.

what I realized is the following:

1. I start reading a new message, I find the lenght of the message, and
I start reading them.
2. however, after a few chars, the END of the message is not
"available" (is not there), and it looks like the bufferedReader simply
complete the message with "space" characters. obviously, my app can't
do much with such message, so it just drops it at some point.
3. finally, the REST of the message arrives, that is, the MISSING PART
of the previous message. having already "read" it (at least that's what
the app thinks), it starts back as if it was a new message, reads the
first 2 characters to check length, come up with an absurd length,
reads much more characters than it should, including ones in the NEXT
messages, well, anyway, it crashes spectacularly

I'd like to find a way to avoid the second point.

how come this guy completes the message with spaces ? why can't it
simply wait for the next TCP packet and proceed ?

I understand I might not have given all the important info about the
app / system, but I'd appreciate if anyone could point me in the right
direction. I've been thinking about timeouts / packet sizes / buffer
length, but I'm not exactly sure how all that is related to this
problem...

anyone has an idea ?

thanks


here's the method that reads on my socket...

private BufferedReader _bufferedReader = new BufferedReader(new
InputStreamReader(_socket.getInputStream(), "ISO-8859-1"));

public void run() {
char[] msg = null;
boolean handle ;

while (!_exit) {
handle = true;
try {
msg = receive();
}
catch (InterruptedIOException iioe) {
handle = false ;
}
catch (IOException ioe) {
_exit = true ;
}

if (!_exit && handle && (msg!=null)) {
_listener.addMessage(msg) ;
}
}

try {
_socket.close() ;
}
catch(Exception e) {
}
}



public char[] receive() throws IOException {
char[] bufLen = new char[2];
try {
_bufferedReader.read(bufLen);
}
catch (SocketException se) {
_exit = true;
se.printStackTrace();
return null;
}

int len = decodeLength(bufLen);

try {
char[] _rawMsg = new char[len];
_bufferedReader.read(_rawMsg);
return _rawMsg;
}
catch (SocketException se) {
se.printStackTrace();
_exit = true ;
}
return null;
}

 

[Matt Humphrey]

Hello,

I'm having a pretty annoying issue with a TCP connection in a
client/server environment.

my client connects to a server that sends it messages of variable
lengths.

on reception of each message, the client strips down the message in the
following manner:

1. the first two characters of the message are first read to indicate
the length of the message (as in the "decodeLength" method in the code
below)
2. knowing the length of the message, I read the appropriate number of
characters on my bufferedReader, and I start over.

try {
char[] _rawMsg = new char[len];
_bufferedReader.read(_rawMsg);

This read operation is not guaranteed to fill the buffer. Depending on how
the transport layer breaks up the original packets, it may return a partial
buffer. You must look at the number of characters returned and keep reading
until you get the whole thing--something like this:

while (len > 0) {
int didRead = _bufferedReader.read(_rawMsg, _rawMsg.length - len, len);
len -= didRead;
}

Cheers,
Matt Humphrey (e-mail address removed) http://www.iviz.com/

 

[John C. Bollinger]

I'm having a pretty annoying issue with a TCP connection in a
client/server environment.

my client connects to a server that sends it messages of variable
lengths.

on reception of each message, the client strips down the message in the
following manner:

1. the first two characters of the message are first read to indicate
the length of the message (as in the "decodeLength" method in the code
below)
2. knowing the length of the message, I read the appropriate number of
characters on my bufferedReader, and I start over.

So far, so good, though you didn't actually include the implementation
of decodeLength(). Your approach is a standard one.

this method has proven effective for a very long time for me, but
recently I installed a new server on a slightly different environment
(stable though), and started experiencing strange things. after
tracking down the errors, I realized that problems arised when TCP
packets were not received properly, and the TCP stack asked for a
resend.

I haven't exactly figured out what's happening at the TCP layer yet,
but I "trust" the TCP stack, and think my client should be able to
handle such situation, however it does not.

[...]

Your client /could/ easily handle the situation, but it is buggy.

public char[] receive() throws IOException {
char[] bufLen = new char[2];
try {
_bufferedReader.read(bufLen);

The above is broken, though in practice it is not likely to fail often.
(See below.)

}
catch (SocketException se) {
_exit = true;
se.printStackTrace();
return null;
}

int len = decodeLength(bufLen);

try {
char[] _rawMsg = new char[len];
_bufferedReader.read(_rawMsg);

This bit has the same bug as the earlier read, but it is more likely to
be affected. (See explanation below.)

return _rawMsg;
}
catch (SocketException se) {
se.printStackTrace();
_exit = true ;
}
return null;
}

You should read the docs of Reader.read(char[]). You will then see that
the method returns an int, which indicates the number of bytes actually
read. This is by no means a formality: the method promises to block
until it receives at least one character or sees the end of the stream,
but it is by no means guaranteed to fill the array, regardless of the
number of characters the sender is sending. There are many reasons why
one logical message might be split up over multiple reads, and the TCP
behavior you observed is just one of them. Using a BufferedReader as
you have done improves the chances of getting a whole array in one read,
but still does not ensure it.

The usual idiom for reading a fixed number of chars from a character
stream is this:

Reader reader;
char[] buf;
int numberToRead;

[Assign suitable values to the variables]

// Read repeatedly until the expected number of chars has been read:
for (int totalCharsRead = 0; totalCharsRead < numberToRead; ) {
int numberLeft = numberToRead - totalCharsRead;
int numberRead = reader.read(buf, totalCharsRead, numberLeft);

if (numberRead < 0) {
// premature end of data
break;
} else {
totalCharsRead += numberRead;
}
}

You can pop that into a handy method so that you don't need to repeat it
every time, but you do need to use something like it for *both* of the
reads in your method. Also, when you do it that way you are effectively
buffering the input manually, so the BufferedReader doesn't provide much
advantage over an unbuffered stream.

By the way, the appropriate technique for reading a fixed number of
bytes from a byte stream is completely parallel to the above.

 

[antoine]

thanks to both of you for the insight, I understand my error...
after running the modified version for several hours, I haven't seen a
single crash (vs one every 30 minutes before...)