NIO, SocketChannel and packet processing

[Qu0ll]

I am using non-blocking NIO with SocketChannel to send data from a server to
a client in "packets". Now, I thought that if I send 3 packets of 1024
bytes each and then call read() at the other end 3 times I would be able to
read 3 separate packets. However, it seems that the underlying protocol
knows nothing about the packet structure so the packets get glued together
and just one read() call on the client may read all 3 packets at once.

Is this correct? If so, how can I delimit packets and have a separate read
for each one? Or is it a case of reading whatever's available and then
breaking it up on the client side using some special byte value?

--
And loving it,

-Q
_________________________________________________
(e-mail address removed)
(Replace the "SixFour" with numbers to email me)

 

[Gordon Beaton]

I am using non-blocking NIO with SocketChannel to send data from a server to
a client in "packets". Now,

I thought that if I send 3 packets of 1024 bytes each and then call
read() at the other end 3 times I would be able to read 3 separate
packets. However, it seems that the underlying protocol knows
nothing about the packet structure so the packets get glued together
and just one read() call on the client may read all 3 packets at
once.

That's normal TCP behaviour (unrelated to java.nio).

Is this correct? If so, how can I delimit packets and have a
separate read for each one? Or is it a case of reading whatever's
available and then breaking it up on the client side using some
special byte value?

If you know that every packet will be N bytes, then always read N byte
packets.

Otherwise, define a packet structure for your application that
includes the packet length at the start of every packet. Start by
reading (say) 4 bytes to get the length, then read again to get the
number of bytes that were specified.

An alternative mechanism is to separate messages with a delimiter, but
that requires you to handle the delimiter specially in those cases
where it may occur within a message. It's also more work for the
recipient to find the delimiter and separate the messages.

/gordon

--

 

[Qu0ll]

Thanks for the helpful reply Gordon.

If you know that every packet will be N bytes, then always read N byte
packets.

The packets are of variable length.

Otherwise, define a packet structure for your application that
includes the packet length at the start of every packet. Start by
reading (say) 4 bytes to get the length, then read again to get the
number of bytes that were specified.

I like this idea. But how do I read a specific number of bytes? Is it a
matter of using a ByteBuffer of the desired size to firstly read the packet
size and then allocating another ByteBuffer with the actual size the same as
the packet size?

An alternative mechanism is to separate messages with a delimiter, but
that requires you to handle the delimiter specially in those cases
where it may occur within a message. It's also more work for the
recipient to find the delimiter and separate the messages.

No, I'll try to get the other system working.

Thanks again.

--
And loving it,

-Q
_________________________________________________
(e-mail address removed)
(Replace the "SixFour" with numbers to email me)

 

[Gordon Beaton]

But how do I read a specific number of bytes? Is it a matter of
using a ByteBuffer of the desired size to firstly read the packet
size and then allocating another ByteBuffer with the actual size the
same as the packet size?

Either that, or use a single ByteBuffer per client that you know will
always be large enough. You can specify both length and offset when
you read in either case.

There will also be times when you only manage to read part of a
message. You need to take note of the return value on each read, and
only return a complete message once the actual number of bytes matches
the expected number for that client's current message.

/gordon

--

 

[Qu0ll]

Either that, or use a single ByteBuffer per client that you know will
always be large enough. You can specify both length and offset when
you read in either case.

I can't see any version of the read() method that allows me to specify how
many bytes to read. I think what you are referring to is the form of the
read() method that allows you to specify which buffer in a buffer array to
use and how many to use.

There will also be times when you only manage to read part of a
message. You need to take note of the return value on each read, and
only return a complete message once the actual number of bytes matches
the expected number for that client's current message.

Well I'm pleased to say that it's now working... almost. It works perfectly
when the server and client are on the same machine. But when I try the
client on a different machine on the LAN, not all the data seems to be
getting through. Surely this cannot be because of transmission errors over
a tiny LAN? How prevalent would such errors be? What hope do I have of
getting it to work over the internet?

--
And loving it,

-Q
_________________________________________________
(e-mail address removed)
(Replace the "SixFour" with numbers to email me)

 

[Gordon Beaton]

I can't see any version of the read() method that allows me to
specify how many bytes to read. I think what you are referring to is
the form of the read() method that allows you to specify which
buffer in a buffer array to use and how many to use.

Sorry, my mistake. Strange that there is no method for reading a
specified number of bytes into a ByteBuffer.

Well I'm pleased to say that it's now working... almost. It works
perfectly when the server and client are on the same machine. But
when I try the client on a different machine on the LAN, not all the
data seems to be getting through. Surely this cannot be because of
transmission errors over a tiny LAN? How prevalent would such errors
be? What hope do I have of getting it to work over the internet?

Since TCP is used for virtually all internet applications, I think
your chances are quite good. I can't say from here exactly what
problem you're having though. Are you checking the return value from
read(), and keeping track of the actual received byte count for each
message?

Try downloading Wireshark to see the actual network traffic your
application is generating.

/gordon

--

 

[Qu0ll]

Since TCP is used for virtually all internet applications, I think
your chances are quite good. I can't say from here exactly what
problem you're having though. Are you checking the return value from
read(), and keeping track of the actual received byte count for each
message?

Try downloading Wireshark to see the actual network traffic your
application is generating.

It's sorted now. The problem was that sometimes fewer bytes than the
expected packet size were being read so now I just make sure I read until
all bytes are processed. This happened more the further the client and
server were apart physically.

Thanks again Gordon.

--
And loving it,

-Q
_________________________________________________
(e-mail address removed)
(Replace the "SixFour" with numbers to email me)

 

[Lew]

The problem was that sometimes fewer bytes than the
expected packet size were being read so now I just make sure I read
until all bytes are processed.

Gordon Beaton had just advised:

 

[Gordon Beaton]

It's sorted now. The problem was that sometimes fewer bytes than the
expected packet size were being read so now I just make sure I read
until all bytes are processed. This happened more the further the
client and server were apart physically.

This thread was starting to sound familiar to me, and it appears you
asked virtually the same questions on Oct 5-6...

Did you not solve this problem *then*?

/gordon

--

 

[Qu0ll]

Gordon Beaton had just advised:

Yes, I was merely pointing out that he was spot on.

--
And loving it,

-Q
_________________________________________________
(e-mail address removed)
(Replace the "SixFour" with numbers to email me)

 

[Qu0ll]

This thread was starting to sound familiar to me, and it appears you
asked virtually the same questions on Oct 5-6...

Did you not solve this problem *then*?

That was with plain-old-standard I/O, not NIO. I have moved everything I
was doing then over to the improved NIO - successfully thankfully.

--
And loving it,

-Q
_________________________________________________
(e-mail address removed)
(Replace the "SixFour" with numbers to email me)