Big Endian - Little Endian

[James Kanze]

I mostly agree, but there are exceptions, and they are not
THAT few. One apparant exception is databases: if you want
high performance and reliability, there is no way out of
writing data explicitly to disk and doing so in a binary
format.

Nobody said to not use a binary format. But a binary format is
not "unformatted". All of the high performance data bases I
know use a proprietary internal format, with little impact on
performance.

Of course, portability suffers but you don't want to port to
exoteric machines anyway.

You might, however, want to be able to reread the data with a
program compiled with a newer version of the compiler. Or with
a program compiled with different compiler options. I've seen
byte order change from one version of the compiler to the next
(Microsoft, for PC's---not the most exoteric machine around),
and all of the compilers I use have options which change the
size of some of the basic types, padding in structures, etc.

 

[James Kanze]

Not if you use an endian neutral file system.

I'm not sure I understand. I always assumed that all file
systems were endian neutral, since they all (the ones I know of,
anyway) manipulate bytes, and nothing else. (This is, in fact,
required by the C and C++ standards: wfstream reads and writes
bytes, not wchar_t.)

And fwrite() (or ostream::write()) definitely just dump the
bits, which only works if you've preformatted the data they are
writing.

 

[Ian Collins]

I'm not sure I understand. I always assumed that all file
systems were endian neutral, since they all (the ones I know of,
anyway) manipulate bytes, and nothing else. (This is, in fact,
required by the C and C++ standards: wfstream reads and writes
bytes, not wchar_t.)

I know, I posted too soon.

The data may be endian neutral, but the file system's own house keeping
data may not be. For example you couldn't take a UFS disk form a Sparc
system and read it on an x86 one.

 

[peter koch]

Nobody said to not use a binary format.  But a binary format is
not "unformatted".  All of the high performance data bases I
know use a proprietary internal format, with little impact on
performance.


You might, however, want to be able to reread the data with a
program compiled with a newer version of the compiler.  Or with
a protgram compiled with different compiler options.

Certainly. This requires you to have some knowledge of the CPU(s) you
target, and you must hope for some sensible strategy from your
compiler vendor.

I do not believe that "plugging the database" right into another kind
of processor is needed. As an example, if the next generation of Intel
processor (or compiler) skipped eight-bit characters entirely, I would
consider this a new architecture and would find it sensible to require
an upgrade from the users (e.g. doing a backup and restore rather than
a raw file copy).

 I've seen
byte order change from one version of the compiler to the next
(Microsoft, for PC's---not the most exoteric machine around),
and all of the compilers I use have options which change the
size of some of the basic types, padding in structures, etc.

You can't in general write a structure in binary. What I am referring
to would be e.g. the layout of a B-tree page, an inode or something
like that.

/Peter

 

[James Kanze]

Certainly. This requires you to have some knowledge of the
CPU(s) you target, and you must hope for some sensible
strategy from your compiler vendor.

My experience has been that that is a rather vain hope. And
realistically, you do expect to use larger integers sooner or
later, especially where file systems are concerned. (The size
of the integers in a data base will in some cases depend on the
size of the largest disk you support.)

I do not believe that "plugging the database" right into
another kind of processor is needed. As an example, if the
next generation of Intel processor (or compiler) skipped
eight-bit characters entirely, I would consider this a new
architecture and would find it sensible to require an upgrade
from the users (e.g. doing a backup and restore rather than a
raw file copy).

Yep, and in order to update and migrate, you need a known and
specified format.

You can't in general write a structure in binary. What I am referring
to would be e.g. the layout of a B-tree page, an inode or something
like that.

The idea that the OS might write the inodes by just dumping the
bits doesn't bother me, even if it is a struct. It's a very
special case. (People programming at that level generally have
to know exactly how the compiler lays things out anyway.) But
most programmers aren't accessing disks (or anything else) at
that level. I'd consider code to write inodes rather rare.

 

[James Kanze]

I know, I posted too soon.

The data may be endian neutral, but the file system's own
house keeping data may not be. For example you couldn't take
a UFS disk form a Sparc system and read it on an x86 one.

At the system level, I can see this. I don't expect to take a
disk from one machine, plug it into another, and have it work
(although there's really no reason why not---we expected it to
work with floppies). In the same vein, I don't expect the
binary image of the system to boot correctly other than on the
system for which it was generated. But how many programs are
writing executables (or even object files) to the system, or are
accessing the low level maintenance data on the disk?

 

[peter koch]

My experience has been that that is a rather vain hope.  And
realistically, you do expect to use larger integers sooner or
later, especially where file systems are concerned.  (The size
of the integers in a data base will in some cases depend on the
size of the largest disk you support.)

Or on the size of the database, you plan to support (considering that
you might allow databases spanning more than one disk).

Yep, and in order to update and migrate, you need a known and
specified format.


The idea that the OS might write the inodes by just dumping the
bits doesn't bother me, even if it is a struct.  It's a very
special case.  (People programming at that level generally have
to know exactly how the compiler lays things out anyway.)  But
most programmers aren't accessing disks (or anything else) at
that level.  I'd consider code to write inodes rather rare.

But then there is also the code from Oracle, Sybase, Ingres and all
the other database-providers - as well as the people writing file-
systems. I agree, that this is quite a specialised domain, but must
probably there are others as well, where it is an advantage to be able
to quickly write data and transfer it to C++ structures (pdf- and jpeg-
files cross my mind).
Still, we both agree that unless you write huge amounts of I/O to
disk, and can profit from a "fast" translation (often, disk i/o times
will dominate the picture), you should do a proper serialisation. And
if you need to do the dirty stuff, you should be very, very careful
and be absolutely certain that you know what you do.

/Peter

 

[Ian Collins]

At the system level, I can see this. I don't expect to take a
disk from one machine, plug it into another, and have it work
(although there's really no reason why not---we expected it to
work with floppies).

If you want to do that, *then* you use an endian neutral file system!

 

[EventHelix.com]

I have this program :

void main()
{
    int i=1;
    if((*(char*)&i)==1)
         printf("The machine is little endian.");
    else
         printf("The machine is big endian.");

}

This program tells me if the machine uses big endian or little endian
format.
But I am not able to understand the working of this program. Can
someone please explain the working.

Thanks in advance.

Thanks,
Niranjan.

The following article explains big and little endian byte ordering:

http://www.eventhelix.com/RealtimeMantra/ByteAlignmentAndOrdering.htm