fstream's write

[aegis]

How can one portably write out to a binary file an integer value
if fstream's write wants a pointer to char read-only?
Or say a float, etc. What if pointer to float or
pointer to int has different alignment requirements than
for a pointer to char read-only? How do we get around
this?

 

[Pavel]

How can one portably write out to a binary file an integer value
if fstream's write wants a pointer to char read-only?
Or say a float, etc. What if pointer to float or
pointer to int has different alignment requirements than
for a pointer to char read-only? How do we get around
this?

My understanding is that you are not writing a pointer to float or char
but you write serialized representation of a float or of a char. Then,
when you read it you have to read it into the properly aligned place
that is aligned correspondently by the compiler. While in the file,
alignment does not matter unless you are going to map it to a memory
area (in which case, the representation can't be made portable). So the
only issue seems to be the data size and the format. Unless you want
compression, you may use Java serialization format (IEEE 754 for single-
and double- precision floats and two's complements format for integers,
8-, 16-, 32- and 64- bit, with MSB first (so called "network" order)).
For example (the below is not exactly by the letter of the Standard, but
will work on many systems),

typedef ... Int32; /* this is system-dependent but you can use int32_t
that is available on many systems (theoretically, it's available in
stdint.h on any C system; in practice, you may have to do some
#ifdef-juggling between stdint.h and inttypes.h depending on the system */

/* the following is a concept only, no error processing, also in
practice you might want to use ntohl, htonl or whatever is available on
your system for efficiency. */

#define INT32_CHARS (32/CHAR_BIT)

void writeInt32(ostream &os, Int32 n) {
unsigned char c[INT32_CHARS];
for (int i = INT32_CHARS; --i > 0
c[(INT32_CHARS - 1) - i] = (unsigned char) (n >>
(CHAR_BIT * i));
c[INT32_CHARS - 1] = (unsigned char) n;
os.write(reinterpret_cast<char *>(&c[0]), 4);
}

void readInt32(istream &os, Int32 &n) {
unsigned char c[INT32_CHARS];
os.read(reinterpret_cast<char *>(&c[0]), 4);
n = 0;
for (int i = 1; i < INT32_CHARS; ++i)
n |= ((UInt32)c[i-1] << (CHAR_BIT * (INT32_CHARS - i)));
n |= ((UInt32)c[INT32_CHARS - 1]);
}

Hope this will help
-Pavel

 

[aegis]

My understanding is that you are not writing a pointer to float or char
  but you write serialized representation of a float or of a char. Then,
when you read it you have to read it into the properly aligned place
that is aligned correspondently by the compiler. While in the file,
alignment does not matter unless you are going to map it to a memory
area (in which case, the representation can't be made portable). So the
only issue seems to be the data size and the format. Unless you want
compression, you may use Java serialization format (IEEE 754 for single-
and double- precision floats and two's  complements format for integers,
8-, 16-, 32- and 64- bit, with MSB first (so called "network" order)).
For example (the below is not exactly by the letter of the Standard, but
will work on many systems),

The problem arises when your write function dereferences the
pointer to char. If the pointer value is preserved from
say pointer to float to pointer to char, then you are not
necessarily guaranteed that the use of the pointer to char
will produce meaningful results.

i.e.,
float f = 3.14f;
float *p = &f;
then someObject.write((const char *)p, ...)

when write attempts to dereference the pointer value,
you enter the realm of unspecified behavior.

So how can this be accomplished portably?

typedef ... Int32; /* this is system-dependent but you can use int32_t
that is available on many systems (theoretically, it's available in
stdint.h on any C system; in practice, you may have to do some
#ifdef-juggling between stdint.h and inttypes.h depending on the system */

/* the following is a concept only, no error processing, also in
practice you might want to use ntohl, htonl or whatever is available on
  your system for efficiency. */

#define INT32_CHARS (32/CHAR_BIT)

void writeInt32(ostream &os, Int32 n) {
         unsigned char c[INT32_CHARS];
         for (int i = INT32_CHARS;  --i > 0
                 c[(INT32_CHARS - 1) - i] = (unsigned char) (n >>
(CHAR_BIT * i));
         c[INT32_CHARS - 1] = (unsigned char) n;
         os.write(reinterpret_cast<char *>(&c[0]), 4);

}

void readInt32(istream &os, Int32 &n) {
         unsigned char c[INT32_CHARS];
         os.read(reinterpret_cast<char *>(&c[0]), 4);
         n = 0;
         for (int i = 1;  i < INT32_CHARS;  ++i)
                 n |= ((UInt32)c[i-1] << (CHAR_BIT * (INT32_CHARS - i)));
         n |= ((UInt32)c[INT32_CHARS - 1]);

}

 

[Brian]

How can one portably write out to a binary file an integer value
if fstream's write wants a pointer to char read-only?
Or say a float, etc.  What if pointer to float or
pointer to int has different alignment requirements than
for a pointer to char read-only?  How do we get around
this?

My understanding is that you are not writing a pointer to float or char
  but you write serialized representation of a float or of a char. Then,
when you read it you have to read it into the properly aligned place
that is aligned correspondently by the compiler. While in the file,
alignment does not matter unless you are going to map it to a memory
area (in which case, the representation can't be made portable). So the
only issue seems to be the data size and the format. Unless you want
compression, you may use Java serialization format (IEEE 754 for single-
and double- precision floats and two's  complements format for integers,
8-, 16-, 32- and 64- bit, with MSB first (so called "network" order)).
For example (the below is not exactly by the letter of the Standard, but
will work on many systems),

typedef ... Int32; /* this is system-dependent but you can use int32_t
that is available on many systems (theoretically, it's available in
stdint.h on any C system; in practice, you may have to do some
#ifdef-juggling between stdint.h and inttypes.h depending on the system */

/* the following is a concept only, no error processing, also in
practice you might want to use ntohl, htonl or whatever is available on
  your system for efficiency. */

#define INT32_CHARS (32/CHAR_BIT)

void writeInt32(ostream &os, Int32 n) {
         unsigned char c[INT32_CHARS];
         for (int i = INT32_CHARS;  --i > 0
                 c[(INT32_CHARS - 1) - i] = (unsigned char) (n >>
(CHAR_BIT * i));
         c[INT32_CHARS - 1] = (unsigned char) n;
         os.write(reinterpret_cast<char *>(&c[0]), 4);

}

I'm not sure why you mention network order. Wouldn't this
function work right on little endian and big endian machines?
The '4' should be INT32_CHARS I guess.


Brian Wood
http://webEbenezer.net

 

[Pavel]

How can one portably write out to a binary file an integer value
if fstream's write wants a pointer to char read-only?
Or say a float, etc. What if pointer to float or
pointer to int has different alignment requirements than
for a pointer to char read-only? How do we get around
this?

My understanding is that you are not writing a pointer to float or char
but you write serialized representation of a float or of a char. Then,
when you read it you have to read it into the properly aligned place
that is aligned correspondently by the compiler. While in the file,
alignment does not matter unless you are going to map it to a memory
area (in which case, the representation can't be made portable). So the
only issue seems to be the data size and the format. Unless you want
compression, you may use Java serialization format (IEEE 754 for single-
and double- precision floats and two's complements format for integers,
8-, 16-, 32- and 64- bit, with MSB first (so called "network" order)).
For example (the below is not exactly by the letter of the Standard, but
will work on many systems),

typedef ... Int32; /* this is system-dependent but you can use int32_t
that is available on many systems (theoretically, it's available in
stdint.h on any C system; in practice, you may have to do some
#ifdef-juggling between stdint.h and inttypes.h depending on the system */

/* the following is a concept only, no error processing, also in
practice you might want to use ntohl, htonl or whatever is available on
your system for efficiency. */

#define INT32_CHARS (32/CHAR_BIT)

void writeInt32(ostream &os, Int32 n) {
unsigned char c[INT32_CHARS];
for (int i = INT32_CHARS; --i > 0
c[(INT32_CHARS - 1) - i] = (unsigned char) (n >>
(CHAR_BIT * i));
c[INT32_CHARS - 1] = (unsigned char) n;
os.write(reinterpret_cast<char *>(&c[0]), 4);

}

I'm not sure why you mention network order.

Just to identify the portable external data format I was going to use
(XDR, essentially).

Wouldn't this

function work right on little endian and big endian machines?

It should work as it is . But ntohl, htonl are often much more
efficient; so if the portability requriements allow, I would use them.

The '4' should be INT32_CHARS I guess.

Correct. Also, INT32_CHARS it could be defined simply as sizeof(Int32).

 

[Pavel]

The problem arises when your write function dereferences the
pointer to char.

In my example of the write function, writeInt32, only a pointer to a
char array element is dereferenced. It was never converted from a
pointer to Int32. With float, getting a portable IEEE 754 representation
in general case is difficult, but does not involve pointers per se (you
will most probably have to play with frexp() function and the likes from
<cmath>).

If, on the other hand, you want to take advantage of a particular
architecture that actually stores floating point numbers in IEEE (754)
(Intel and Sparc CPUs do, for example), you will have to get your hands
dirty and learn what really happens to pointer to float when it gets
converted to pointer to char for every particular platform you want to
support (the good news is that the answer for the above two and probably
many others will be "nothing" AFAIK). Certainly any useful
implementation will use this second approach. You will have to write
some #ifdefs; it's ok, portable code rarely looks "clean".

BTW: If your question is for a practical purpose and you do not require
top-notch performance, you might want to figure out if there is an
implementation of xdr (RFC1832) covering all your platforms of interest
and see if you can just use it instead of re-inventing this particular
wheel.

-Pavel


If the pointer value is preserved from

say pointer to float to pointer to char, then you are not
necessarily guaranteed that the use of the pointer to char
will produce meaningful results.

i.e.,
float f = 3.14f;
float *p = &f;
then someObject.write((const char *)p, ...)

when write attempts to dereference the pointer value,
you enter the realm of unspecified behavior.

So how can this be accomplished portably?

typedef ... Int32; /* this is system-dependent but you can use int32_t
that is available on many systems (theoretically, it's available in
stdint.h on any C system; in practice, you may have to do some
#ifdef-juggling between stdint.h and inttypes.h depending on the system */

/* the following is a concept only, no error processing, also in
practice you might want to use ntohl, htonl or whatever is available on
your system for efficiency. */

#define INT32_CHARS (32/CHAR_BIT)

void writeInt32(ostream &os, Int32 n) {
unsigned char c[INT32_CHARS];
for (int i = INT32_CHARS; --i > 0
c[(INT32_CHARS - 1) - i] = (unsigned char) (n >>
(CHAR_BIT * i));
c[INT32_CHARS - 1] = (unsigned char) n;
os.write(reinterpret_cast<char *>(&c[0]), 4);

}

void readInt32(istream &os, Int32 &n) {
unsigned char c[INT32_CHARS];
os.read(reinterpret_cast<char *>(&c[0]), 4);
n = 0;
for (int i = 1; i < INT32_CHARS; ++i)
n |= ((UInt32)c[i-1] << (CHAR_BIT * (INT32_CHARS - i)));
n |= ((UInt32)c[INT32_CHARS - 1]);

}