ctypes in python failed to honor c_int

[Jerry Fleming]

Hi,

I have a binary file written with c structures. Each record contains a
null-terminated string followed by two 4-bytes integers. I wrote a small
segment of python code to parse this file in this way:
[coe]
#!/usr/bin/python

from ctypes import *

class Entry(Structure):
_fields_ = ('w', c_char_p), ('s', c_uint, 32), ('l', c_uint, 32)

idx = open('x.idx', 'rb')
str = idx.read(1000)
obj = Entry(str)
print obj.w
print obj.s
print obj.l
[/code]
where the field w is the string, and s and l are the integers. Problem
is that, I can only get the strings, not the integers. Well, I did got
integers, but they are all zeros. What should I do to get the real numbers?

Thanks for help.

- Jerry

 

[Gabriel Genellina]

I have a binary file written with c structures. Each record contains a
null-terminated string followed by two 4-bytes integers. I wrote a small
segment of python code to parse this file in this way:
[coe]
#!/usr/bin/python

from ctypes import *

class Entry(Structure):
        _fields_ = ('w', c_char_p), ('s', c_uint, 32), ('l', c_uint, 32)

idx = open('x.idx', 'rb')
str = idx.read(1000)
obj = Entry(str)
print obj.w
print obj.s
print obj.l
[/code]
where the field w is the string, and s and l are the integers. Problem
is that, I can only get the strings, not the integers. Well, I did got
integers, but they are all zeros. What should I do to get the real numbers?

So the string has a variable length? For "Hello" you have
'h','e','l','l','o', a zero byte, followed by the two integers? This
is somewhat unusual for a C struct (in fact you can't declare it in
C). Perhaps the string is actually a char[n] array with a declared
maximum size?

Try printing repr(a_few_read_bytes) or a_few_read_bytes.encode("hex")
to see the actual file contents.

 

[Jerry Fleming]

I have a binary file written with c structures. Each record contains a
null-terminated string followed by two 4-bytes integers. I wrote a small
segment of python code to parse this file in this way:
[coe]
#!/usr/bin/python

from ctypes import *

class Entry(Structure):
_fields_ = ('w', c_char_p), ('s', c_uint, 32), ('l', c_uint, 32)

idx = open('x.idx', 'rb')
str = idx.read(1000)
obj = Entry(str)
print obj.w
print obj.s
print obj.l
[/code]
where the field w is the string, and s and l are the integers. Problem
is that, I can only get the strings, not the integers. Well, I did got
integers, but they are all zeros. What should I do to get the real numbers?

So the string has a variable length? For "Hello" you have
'h','e','l','l','o', a zero byte, followed by the two integers? This
is somewhat unusual for a C struct (in fact you can't declare it in
C). Perhaps the string is actually a char[n] array with a declared
maximum size?

Yes, it has a variable length. The C version of the structure is
something like this:

struct entry {
     char *str,
     int start,
     int length
}

And adding repr() would print something like this:

'(as) mad as a 
hatter\x00\x00\x00\x00\x00\x00\x00\x00\x1f-ish\x00\x00\x00\x00\x1f\x00\x00\x00@-ism\x00\x00\x00\x00_\x00\x00\x00B-ist\x00\x00\x00\x00\xa1\x00\x00\x00J.AU\x00\x00\x00\x00\xeb\x00\x00\x00P.EXE\x00\x00\x00\x01;\x00\x00\x00`.GIF\x00\x00\x00'
(as) mad as a hatter
0
0

where the first line is the result of repr(). We can find that, after
the null-terminated string '(as) mad as a hatter', there are two
integers, 0 and 31 (0x1f). But python treat 31 as zero.

 

[Gabriel Genellina]

I have a binary file written with c structures. Each record contains a
null-terminated string followed by two 4-bytes integers. I wrote a small
segment of python code to parse this file in this way:
[coe]
#!/usr/bin/python
from ctypes import *
class Entry(Structure):
        _fields_ = ('w', c_char_p), ('s', c_uint, 32), ('l', c_uint, 32)
idx = open('x.idx', 'rb')
str = idx.read(1000)
obj = Entry(str)
print obj.w
print obj.s
print obj.l
[/code]
where the field w is the string, and s and l are the integers. Problem
is that, I can only get the strings, not the integers. Well, I did got
integers, but they are all zeros. What should I do to get the real numbers?

So the string has a variable length? For "Hello" you have
'h','e','l','l','o', a zero byte, followed by the two integers? This
is somewhat unusual for a C struct (in fact you can't declare it in
C). Perhaps the string is actually a char[n] array with a declared
maximum size?

Yes, it has a variable length. The C version of the structure is
something like this:

struct entry {
     char *str,
     int start,
     int length}

But this doesn't match the file contents. There are no pointers in the
file.

And adding repr() would print something like this:

'(as) mad as a
hatter\x00\x00\x00\x00\x00\x00\x00\x00\x1f-ish\x00\x00\x00\x00\x1f\x00\x00\­x00@-ism\x00\x00\x00\x00_\x00\x00\x00B-ist\x00\x00\x00\x00\xa1\x00\x00\x00J­.AU\x00\x00\x00\x00\xeb\x00\x00\x00P.EXE\x00\x00\x00\x01;\x00\x00\x00`..GIF\­x00\x00\x00'
(as) mad as a hatter
0
0

where the first line is the result of repr(). We can find that, after
the null-terminated string '(as) mad as a hatter', there are two
integers, 0 and 31 (0x1f). But python treat 31 as zero.

Ah, but it doesn't "treat 31 as zero". Entry(str) is the same as
Entry(w=str), that is, you are initializing the w attribute alone,
leaving the other two integers as 0.
I don't know how to use ctypes to read the structure (nor if it is
possible at all), I would read it normally with Python code and build
the struct afterwards (in case it is used to call any C code).

w, data = data.split('\x00', 1)
s, l = struct.unpack("ll", data[:8])
data= data[8:]

 

[Jerry Fleming]

I have a binary file written with c structures. Each record contains a
null-terminated string followed by two 4-bytes integers. I wrote a small
segment of python code to parse this file in this way:
[coe]
#!/usr/bin/python
from ctypes import *
class Entry(Structure):
_fields_ = ('w', c_char_p), ('s', c_uint, 32), ('l', c_uint, 32)
idx = open('x.idx', 'rb')
str = idx.read(1000)
obj = Entry(str)
print obj.w
print obj.s
print obj.l
[/code]
where the field w is the string, and s and l are the integers. Problem
is that, I can only get the strings, not the integers. Well, I did got
integers, but they are all zeros. What should I do to get the real numbers?
So the string has a variable length? For "Hello" you have
'h','e','l','l','o', a zero byte, followed by the two integers? This
is somewhat unusual for a C struct (in fact you can't declare it in
C). Perhaps the string is actually a char[n] array with a declared
maximum size?

Yes, it has a variable length. The C version of the structure is
something like this:

struct entry {
char *str,
int start,
int length}

But this doesn't match the file contents. There are no pointers in the
file.

And adding repr() would print something like this:

'(as) mad as a
hatter\x00\x00\x00\x00\x00\x00\x00\x00\x1f-ish\x00\x00\x00\x00\x1f\x00\x00\­x00@-ism\x00\x00\x00\x00_\x00\x00\x00B-ist\x00\x00\x00\x00\xa1\x00\x00\x00J­.AU\x00\x00\x00\x00\xeb\x00\x00\x00P.EXE\x00\x00\x00\x01;\x00\x00\x00`.GIF\­x00\x00\x00'
(as) mad as a hatter
0
0

where the first line is the result of repr(). We can find that, after
the null-terminated string '(as) mad as a hatter', there are two
integers, 0 and 31 (0x1f). But python treat 31 as zero.

Ah, but it doesn't "treat 31 as zero". Entry(str) is the same as
Entry(w=str), that is, you are initializing the w attribute alone,
leaving the other two integers as 0.
I don't know how to use ctypes to read the structure (nor if it is
possible at all), I would read it normally with Python code and build
the struct afterwards (in case it is used to call any C code).

w, data = data.split('\x00', 1)
s, l = struct.unpack("ll", data[:8])
data= data[8:]

Oh yes. What an idiot I am. Thanks Gabriel very much.