convert byte array to hex string using BigInteger

[Laura Schmidt]

Hi,

I try to convert a byte array to a hex string like this:

private static String hex_encode (byte [] val)
{
BigInteger b = new BigInteger(val);

String t = b.toString(16);

return (t);
}

For a long byte array it returns a "negative" hex string, i. e. starting
with a "-" sign.

But I want just the bytes in the array converted to hex representation,
each one ranging from "00" to "FF". There should be no minus sign then.

Can you help?
Thanks!

Laura

 

[Eric Sosman]

Hi,

I try to convert a byte array to a hex string like this:

private static String hex_encode (byte [] val)
{
BigInteger b = new BigInteger(val);

String t = b.toString(16);

return (t);
}

For a long byte array it returns a "negative" hex string, i. e. starting
with a "-" sign.

But I want just the bytes in the array converted to hex representation,
each one ranging from "00" to "FF". There should be no minus sign then.

Can you help?

// Untested:
private static String hexEncode(byte[] val) {
StringBuilder buff = new StringBuilder(2 * val.length);
for (byte b : val) {
buff.append("0123456789ABCDEF".charAt((b >> 4) & 0xF));
buff.append("0123456789ABCDEF".charAt(b & 0xF));
}
return buff.toString();
}

There are other ways, too.

 

[Laura Schmidt]

This is a case of RTFM. The Javadocs for BigInteger(byte[]) tell you
that it expects the byte array in two's complement representation, so
a leading 1 bit is interpreted as a negative number. You need a
different constructor:

public BigInteger(int signum, byte[] magnitude)

You are right, sorry.

Now I get a hex string without sign. But now the decode method does not
return the original bytes anymore:

private byte [] hex_decode (String val)
{
BigInteger b = new BigInteger (val,16);
byte [] t = b.toByteArray();

return (t);
}

There is no other constructor for byte arrays.
And I don't really understand why it doesn't return the original byte array.

Laura

 

[markspace]

public static void main(String[] args) {
byte a = -1; //0xFF
byte b = 127; //0x7F
byte c = -128; //0x80
byte d = 64; //0x40
byte e = 16; //0x10
byte f = 99; //0x5A

Laura, if you're still reading, please note especially this part of the
sample program. If you want to know if your idea is going to work, you
have to test it. This list of test vectors is excellent, and also an
excellent example of how to unit test a method.

You'll need to do testing like this not only for yourself, now, to
assure you that your method really works, but also for posterity, by
which I mean also yourself, but three months from now when you have
completely forgotten the details of any method you implement today.

(Or if you're like me, posterity means about two weeks from now. Or less.)

 

[Joerg Meier]

I try to convert a byte array to a hex string like this:

private static String hex_encode (byte [] val)
{
BigInteger b = new BigInteger(val);

String t = b.toString(16);

return (t);
}

For a long byte array it returns a "negative" hex string, i. e. starting
with a "-" sign.

But I want just the bytes in the array converted to hex representation,
each one ranging from "00" to "FF". There should be no minus sign then.

Can you help?

If you are not married to reinventing the wheel, such as by a homework
assignment or whatever, you could always use:

DatatypeConverter.printHexBinary(val);

instead.

Liebe Gruesse,
Joerg

 

[Lew]

This is a case of RTFM. The Javadocs for BigInteger(byte[]) tell you
that it expects the byte array in two's complement representation, so
a leading 1 bit is interpreted as a negative number. You need a
different constructor:

public BigInteger(int signum, byte[] magnitude)

You are right, sorry.

Now I get a hex string without sign. But now the decode method does not
return the original bytes anymore:

private byte [] hex_decode (String val)
{
BigInteger b = new BigInteger (val,16);
byte [] t = b.toByteArray();

return (t);
}

There is no other constructor for byte arrays.
And I don't really understand why it doesn't return the original byte
array.

I'm not sure what all this twos compliment stuff is about

That's spelled "two's-complement".

twos compliment is just a number representation scheme

And the purpose of that constructor is just to convert a representation
in that scheme to the correct number.

There should be no need to 'convert' anything

If there isn't, then don't use that constructor.

 

[Roedy Green]

Here is a low level direct way to convert byte[] to a hex String. It
should be quite a bit faster than using high level routines.

/**
* Fast convert a byte array to a hex string
* with possible leading zero.
* @param b array of bytes to convert to string
* @return hex representation, two chars per byte.
*/
public static String toHexString ( byte[] b )
{
StringBuilder sb = new StringBuilder( b.length * 2 );
for ( int i=0; i<b.length; i++ )
{
// look up high nibble char
sb.append( hexChar [( b & 0xf0 ) >>> 4] );

// look up low nibble char
sb.append( hexChar [b & 0x0f] );
}
return sb.toString();
}

/**
* table to convert a nibble to a hex char.
*/
static char[] hexChar = {
'0' , '1' , '2' , '3' ,
'4' , '5' , '6' , '7' ,
'8' , '9' , 'a' , 'b' ,
'c' , 'd' , 'e' , 'f'};

 

[Joerg Meier]

Here is a low level direct way to convert byte[] to a hex String. It
should be quite a bit faster than using high level routines.

/**
* Fast convert a byte array to a hex string
* with possible leading zero.
* @param b array of bytes to convert to string
* @return hex representation, two chars per byte.
*/
public static String toHexString ( byte[] b )
{
StringBuilder sb = new StringBuilder( b.length * 2 );
for ( int i=0; i<b.length; i++ )
{
// look up high nibble char
sb.append( hexChar [( b & 0xf0 ) >>> 4] );

// look up low nibble char
sb.append( hexChar [b & 0x0f] );
}
return sb.toString();
}

Note that this is pretty much a copy and paste version of the converter
that already comes with the JRE:

from javax.xml.bind.DatatypeConverterImpl:

public String printHexBinary(byte[] data) {
StringBuilder r = new StringBuilder(data.length * 2);
for (byte b : data) {
r.append(hexCode[(b >> 4) & 0xF]);
r.append(hexCode[(b & 0xF)]);
}
return r.toString();
}

For a reason I was unable to discern it was actually a teensy bit slower
than the Oracle implementation.

Liebe Gruesse,
Joerg

 

[Roedy Green]

For a reason I was unable to discern it was actually a teensy bit slower
than the Oracle implementation.

I forgot to make my static array final. That might have done it.
The corrected version is posted at http://mindprod.com/jgloss/hex.html

How did you think to look for such a method in that package by that
name?

 

[Roedy Green]

For a reason I was unable to discern it was actually a teensy bit slower
than the Oracle implementation.

The code looks almost identical. Probably the difference was caused by
different "warmup" times before starting to measure. You have to give
it time to apply the hotspot.

 

[Joerg Meier]

I forgot to make my static array final. That might have done it.
The corrected version is posted at http://mindprod.com/jgloss/hex.html

It has not made any difference in my test case, but I will readily admit
that I only jotted down a very primitive one, so it's entirely possible
that the issue was on my end. Either way, it was only around 5-10%, so I
didn't bother to look into it too much.

How did you think to look for such a method in that package by that
name?

I must admit that I don't recall how I first came upon it, but ever since I
did, I retained to memory that Java "has that", and the few times I needed
it, I just used Google until I found "the one in the JRE". It's not the
most obvious of places, I agree.

Liebe Gruesse,
Joerg

 

[Joerg Meier]

The code looks almost identical. Probably the difference was caused by
different "warmup" times before starting to measure. You have to give
it time to apply the hotspot.

I looped both versions multiple times, and while times varied, yours was
consistently slower. But, like I said in my other post, not nearly by
enough as to prefer one over the other.

For what it's worth, I just noticed that my code is from OpenJDK, but I
tested the code from Oracle. Is there still a difference with these
classes, or are they identical now ? I'm not near my dev PC with the Oracle
JDK/sources right now.

Liebe Gruesse,
Joerg

 

[Arne Vajhøj]

I try to convert a byte array to a hex string like this:

private static String hex_encode (byte [] val)
{
BigInteger b = new BigInteger(val);

String t = b.toString(16);

return (t);
}

For a long byte array it returns a "negative" hex string, i. e. starting
with a "-" sign.

But I want just the bytes in the array converted to hex representation,
each one ranging from "00" to "FF". There should be no minus sign then.

You have already received several working solutions.

I will strongly recommend you drop the idea about using
BigInteger for this.

It becomes extremely tricky to get it right - you have already
found the sign problem - next problem will be the leading zero
problem.

See below for coding examples (and I am not even sure that I got the
BigInteger hack correct).

Arne

====

import java.math.BigInteger;
import java.util.Arrays;

import javax.xml.bind.DatatypeConverter;

public class ToHex {
private static void test(byte[] ba, ByteArrayHex cvt) {
String s = cvt.encode(ba);
System.out.println(s);
byte[] ba2 = cvt.decode(s);
if(ba2.length != ba.length) {
System.out.println("Length does not match: " + ba2.length + " != " +
ba.length);
}
for(int i = 0; i < Math.min(ba2.length, ba.length); i++) {
if(ba2 != ba) {
System.out.println("Byte " + i + " does not match: " + ba2 + " !=
" + ba);
}
}
}
private static void testAll(byte[] ba) {
test(ba, new ManualWithInteger());
test(ba, new ManualWithString());
test(ba, new ManualWithCalc());
test(ba, new JAXBTrick());
test(ba, new BigIntegerHack());
}
public static void main(String[] args) {
testAll(new byte[] { 1, 2, 3, 4, 5 });
testAll(new byte[] { -1, -2, -3, -4, -5 });
testAll(new byte[] { 0, 0, 0 });
}
}

interface ByteArrayHex {
public String encode(byte[] ba);
public byte[] decode(String s);
}

class ManualWithInteger implements ByteArrayHex {
public String encode(byte[] ba) {
StringBuilder sb = new StringBuilder();
for(int i = 0; i < ba.length; i++) {
sb.append(Integer.toHexString((ba >> 4) & 0x0F));
sb.append(Integer.toHexString(ba & 0x0F));
}
return sb.toString();
}
public byte[] decode(String s) {
int n = s.length() / 2;
byte[] res = new byte[n];
for(int i = 0; i < n; i++) {
res = (byte)(Integer.parseInt(s.substring(2 * i, 2 * i + 2),
16));
}
return res;
}
}

class ManualWithString implements ByteArrayHex {
private static final String HEX = "0123456789abcdef";
public String encode(byte[] ba) {
StringBuilder sb = new StringBuilder();
for(int i = 0; i < ba.length; i++) {
sb.append(HEX.charAt((ba >> 4) & 0x0F));
sb.append(HEX.charAt(ba & 0x0F));
}
return sb.toString();
}
public byte[] decode(String s) {
int n = s.length() / 2;
byte[] res = new byte[n];
for(int i = 0; i < n; i++) {
res = (byte)(HEX.indexOf(s.charAt(2 * i)) << 4 |
HEX.indexOf(s.charAt(2 * i + 1)));
}
return res;
}
}

class ManualWithCalc implements ByteArrayHex {
private char encode(int v) {
if(v < 10) {
return (char)('0' + v);
} else if(v < 16) {
return (char)('a' + v - 10);
} else {
throw new RuntimeException("Invalid hex value");
}
}
public String encode(byte[] ba) {
StringBuilder sb = new StringBuilder();
for(int i = 0; i < ba.length; i++) {
sb.append(encode((ba >> 4) & 0x0F));
sb.append(encode(ba & 0x0F));
}
return sb.toString();
}
private int decode(char c) {
if('0' <= c && c <= '9') {
return c - '0';
} else if('a' <= c && c <= 'f') {
return c - 'a' + 10;
} else {
throw new RuntimeException("Invalid hex value");
}
}
public byte[] decode(String s) {
int n = s.length() / 2;
byte[] res = new byte[n];
for(int i = 0; i < n; i++) {
res = (byte)(decode(s.charAt(2 * i)) << 4 |
decode(s.charAt(2 * i + 1)));
}
return res;
}
}

class JAXBTrick implements ByteArrayHex {
public String encode(byte[] ba) {
return DatatypeConverter.printHexBinary(ba);
}
public byte[] decode(String s) {
return DatatypeConverter.parseHexBinary(s);
}
}

class BigIntegerHack implements ByteArrayHex {
private String leftPad(String s, char c, int totlen) {
StringBuilder sb = new StringBuilder();
for(int i = s.length(); i < totlen; i++) sb.append(c);
sb.append(s);
return sb.toString();
}
private byte[] leftTrim(byte[] ba, int len) {
return len < ba.length ? Arrays.copyOfRange(ba, ba.length - len,
ba.length) : ba;
}
private byte[] leftPad(byte[] ba, byte b, int totlen) {
byte[] res = new byte[totlen];
int padlen = totlen - ba.length;
for(int i = 0; i < padlen; i++) res = b;
System.arraycopy(ba, 0, res, padlen, ba.length);
return res;
}
public String encode(byte[] ba) {
BigInteger bi = new BigInteger(1, ba);
String s = bi.toString(16);
return leftPad(s, '0', ba.length * 2);
}
public byte[] decode(String s) {
BigInteger bi = new BigInteger(s, 16);
return leftPad(leftTrim(bi.toByteArray(), s.length() / 2), (byte)0,
s.length() / 2);
}
}

 

[Arne Vajhøj]

This is a case of RTFM. The Javadocs for BigInteger(byte[]) tell you
that it expects the byte array in two's complement representation, so
a leading 1 bit is interpreted as a negative number. You need a
different constructor:

public BigInteger(int signum, byte[] magnitude)

You are right, sorry.

Now I get a hex string without sign. But now the decode method does not
return the original bytes anymore:

private byte [] hex_decode (String val)
{
BigInteger b = new BigInteger (val,16);
byte [] t = b.toByteArray();

return (t);
}

There is no other constructor for byte arrays.
And I don't really understand why it doesn't return the original byte
array.

I'm not sure what all this twos compliment stuff is about
twos compliment is just a number representation scheme
There should be no need to 'convert' anything

The following program works with the following observation

If the most significant byte is positive e.g >= 1 && <= 127 or 0x7F
the conversion works both ways

If the most significant byte is negative or 0 e.g <= 0 && >= -128 or
0x80 then the conversion works with one proviso

The contract for BigInteger#toByteArray() contains the following text

"The array will contain the minimum number of bytes required to
represent this BigInteger, including at least one sign bit"

so, if the MSB is negative there will be an additional byte in the MSG
position after calling toByteArray set to 0 which indicates that the
following is a positive number, if the MSB is positive there is no need
for an additional byte as the first bit in the MSB is 0 thereby marking
the following number as positive ... interesting, never seen this
before... apart from that it seems to work.

You will also get into problems with leading zero bytes.

Arne

 

[Arne Vajhøj]

I try to convert a byte array to a hex string like this:

private static String hex_encode (byte [] val)
{
BigInteger b = new BigInteger(val);

String t = b.toString(16);

return (t);
}

For a long byte array it returns a "negative" hex string, i. e. starting
with a "-" sign.

But I want just the bytes in the array converted to hex representation,
each one ranging from "00" to "FF". There should be no minus sign then.

You have already received several working solutions.

I will strongly recommend you drop the idea about using
BigInteger for this.

It becomes extremely tricky to get it right - you have already
found the sign problem - next problem will be the leading zero
problem.

One of the problems is that toString(radix) and toHexString()
are not the same method.

Demo:

public class SimpleHex {
public static void main(String[] args) {
for(int i = -1; i <= 1; i++) {
System.out.println(Integer.toString(i, 16));
System.out.println(Integer.toHexString(i));
}
}
}


Arne

 

[Laura Schmidt]

On 6/20/2013 8:03 AM, Laura Schmidt wrote:

I will strongly recommend you drop the idea about using
BigInteger for this.

Thank you! I agree. In general, I try not to duplicate code that already
exists in standard libraries, but in this case there were too many hacks
needed when using BigInteger.

Laura

 

[Joerg Meier]

Thank you! I agree. In general, I try not to duplicate code that already
exists in standard libraries, but in this case there were too many hacks
needed when using BigInteger.

Thankfully, as I pointed out in
<[email protected]>, you still don't need to
duplicate code that already exists.

Liebe Gruesse,
Joerg

 

[Stanimir Stamenkov]

Wed, 26 Jun 2013 22:12:30 -0400, /Arne Vajhøj/:

You have already received several working solutions.

I will strongly recommend you drop the idea about using
BigInteger for this.

It becomes extremely tricky to get it right - you have already
found the sign problem - next problem will be the leading zero
problem.

[... bunch of code...]

I may be dense, but I didn't got why using BigInteger is not good?
What's wrong with the following, for example:

byte[] val;
...
String.format("%064x", new BigInteger(1, val));

I guess one could come up with a method like:

String toHexString(byte[] val, int len) {
BigInteger num = new BigInteger(1, val);
return (len > 0) ? String.format("%0" + len + "x", num)
: String.format("%x", num);
}

Doesn't it appear sufficient?

 

[Arne Vajhøj]

Wed, 26 Jun 2013 22:12:30 -0400, /Arne Vajhøj/:

You have already received several working solutions.

I will strongly recommend you drop the idea about using
BigInteger for this.

It becomes extremely tricky to get it right - you have already
found the sign problem - next problem will be the leading zero
problem.

[... bunch of code...]

I may be dense, but I didn't got why using BigInteger is not good?
What's wrong with the following, for example:

byte[] val;
...
String.format("%064x", new BigInteger(1, val));

You will not be able to get back to the same byte array with that.

I guess one could come up with a method like:

String toHexString(byte[] val, int len) {
BigInteger num = new BigInteger(1, val);
return (len > 0) ? String.format("%0" + len + "x", num)
: String.format("%x", num);
}

Doesn't it appear sufficient?

It seems to work.

I would probably drop the len argument and use 2*val.length.

But it is still not as simple as the DatatypeConverter.

Arne

 

[aydin.k.abadi]

Hi
I`m wondering, how I can implement java constructor in C++ and what private member I need ?
Thanks Aydin