Storing binary data in String/StringBuffer

[vj]

Hi,

I am a java new bie and facing some issues regarding storing binary
data in string buffer. Is it really safe to store binary data in
StringBuffers since i need to search for a binary sequence '\r\n' and
Stringbuffer makes it easy via its indexOf method.

If stringbuffer is not the safe place then where should i store binary
data except fixed size byte arrays..

Thanks,

Regards,
VJ

 

[Lew]

Hi,

I am a java new bie and facing some issues regarding storing binary
data in string buffer. Is it really safe to store binary data in
StringBuffers since i need to search for a binary sequence '\r\n' [sic] and
Stringbuffer makes it easy via its indexOf method.

If stringbuffer is not the safe place then where should i store binary
data except fixed size byte arrays..

<http://java.sun.com/javase/6/docs/api/java/io/ByteArrayOutputStream.html>

String is a Bad Thing for binary data - it is a multibyte-character encoded
format.

"\r\n" is not a binary sequence, it's a character sequence.

-- Lew

 

[Oliver Wong]

Hi,

I am a java new bie and facing some issues regarding storing binary
data in string buffer. Is it really safe to store binary data in
StringBuffers since i need to search for a binary sequence '\r\n' and
Stringbuffer makes it easy via its indexOf method.

The StringBuffer itself is safe, in the sense that it won't
mysteriously throw away your data, or broadcast it all over the Internet;
It's the part where you decode/encode from binary data to character data
that's dangerous, in the sense that you'll probably get it wrong (this
isn't a slight against you; almost everybody will get it wrong, myself
included).

If stringbuffer is not the safe place then where should i store binary
data except fixed size byte arrays..

Maybe java.nio.ByteBuffer? It'd be easier to say if you could explain
why fixed sized byte arrays are not suitable for your task.

- Oliver

 

[vj]

Hi,

I am builing network server whoese protocol is line oriented.
Currently i am using nio and a problem with fixed bufferes is that
they are cannot handle lines greater then a certain treshhold. hower
my client can send me an arbitary long line. So presently i receive
the data from nio byte buffer and push it into a string buffer. then i
search for "\r\n" and slice out that section and
do the parsing.

this solution worked fine for text data. however as soon as i switched
to brinary data it started changing the bytes received to some other
bytes!! for example when i received ascii 0x40 it converts it to
'?' !!!

I read some where on the net that its best to handle binary by byte[]
converting it to string and then back to byte[] is risky.

If i should not use StringBuffer then what else should i use.

Thanks,
VJ

 

[Oliver Wong]

Hi,

I am builing network server whoese protocol is line oriented.
Currently i am using nio and a problem with fixed bufferes is that
they are cannot handle lines greater then a certain treshhold. hower
my client can send me an arbitary long line. So presently i receive
the data from nio byte buffer and push it into a string buffer. then i
search for "\r\n" and slice out that section and
do the parsing.

this solution worked fine for text data. however as soon as i switched
to brinary data it started changing the bytes received to some other
bytes!! for example when i received ascii 0x40 it converts it to
'?' !!!

I read some where on the net that its best to handle binary by byte[]
converting it to string and then back to byte[] is risky.

If i should not use StringBuffer then what else should i use.

If you are in control of the protocol, then you should redesign it, as
you are trying to apply a "line oriented protocol" (where the concept of a
"line" is inherently text-based) to binary data.

If you are not in control of the protocol, find out who is, and ask
them how you are expected to send binary data through it.

- Oliver

 

[Lew]

<http://java.sun.com/javase/6/docs/api/java/io/ByteArrayOutputStream.html>

String is a Bad Thing for binary data - it is a multibyte-character encoded
format.

"\r\n" is not a binary sequence, it's a character sequence.

-- Lew

 

[Chris Uppal]

If you are in control of the protocol, then you should redesign it, as
you are trying to apply a "line oriented protocol" (where the concept of a
"line" is inherently text-based) to binary data.

Text-oriented protocols can work well, even if used for binary data (consider
HTTP for one example). The trick is to encode the binary data as text before
transmission; one such encoding, which is widely used and understood, is
"Base64" encoding. Obviously both parties to the transmission have to agree on
what encoding is in use, and on which parts of the data are so encoded.

-- chris

 

[vj]

Text-oriented protocols can work well, even if used for binary data (consider
HTTP for one example). The trick is to encode the binary data as text before
transmission; one such encoding, which is widely used and understood, is
"Base64" encoding. Obviously both parties to the transmission have to agree on
what encoding is in use, and on which parts of the data are so encoded.

-- chris

actually the protocol is not entirly lineoriented. The commands are
delimited by new line charectets but the data is handled as it is.

For instance there is a command AddImage 6481
This indicates that the client is going to send an image of size 6481
bytes. After receiving the command the next 6481 bytes that the client
sends will be not interpretted and are stored as it in a file.

The problem is occuring due to the way i am handing this scenario.
Presently i use a stringbuffer to buffer all this data. partly because
finding the length of stored data in ByteArrayOutputStream class is
inefficient ( you need to call toByteArray().length, that will create
a new new copy!!).
In stringbuffer you simply need to call .length() and we have the
length. And i think thats where the problem lies.

Redesigning the protocol is an option but i would like to avoid it as
much as possible.

 

[vj]

I found yet another thread
http://groups.google.com/group/comp.lang.java.programmer/browse_thread/thread/b382e6a714a73f90
in this group that is describing problem similar to me. Here you guys
have said that when storing binary data in a string/stringbuffer you
need to be sure abt the charecter-set that has being used.

 

[Piotr Kobzda]

The problem is occuring due to the way i am handing this scenario.
Presently i use a stringbuffer to buffer all this data. partly because
finding the length of stored data in ByteArrayOutputStream class is
inefficient ( you need to call toByteArray().length, that will create
a new new copy!!).
In stringbuffer you simply need to call .length() and we have the
length. And i think thats where the problem lies.

Thus familiarize yourself with size() method from ByteArrayOutputStream.
It should solve the problem.


piotr

 

[Chris Uppal]

actually the protocol is not entirly lineoriented. The commands are
delimited by new line charectets but the data is handled as it is.

For instance there is a command AddImage 6481
This indicates that the client is going to send an image of size 6481
bytes. After receiving the command the next 6481 bytes that the client
sends will be not interpretted and are stored as it in a file.

As I think Oliver has already said, a protocol which mixes text and binary is
awkwardly defined. It isn't /too/ difficult to deal with the combination, but
it's more work than I guess you are expecting.

If you want to make it simple for yourself (or rather, make it as simple as the
text/binary combo ever gets), then treat the input as /binary/. You can
extract text data from a binary stream much more easily than you can extract
binary from text stream.

Read the input as binary, looking for sequences terminated by the binary
equivalent of \r\n (whatever that might be in the character encoding that the
sender is using -- almost certainly 0xD 0xA). Then convert that to text
(again, using whatever character encoding the sender is using -- you can
probably assume US-ASCII), and decode the interpret the resulting string in the
way you already are doing. When a command announces that 6481 of binary data
will be the next thing to arrive, you don't need to do anything special, just
consume the next 6481 bytes of input.

If you don't do that, then the binary->text conversion (in the java.io.Reader)
will have converted the binary data into text, and the best that can happen
then is that you have to work out how to reverse that transformation. You
/might/ be able to do that (depending on the encoding, it can be fairly easy or
totally impossible) but it'll probably be difficult and will almost certainly
be inefficient.

-- chris