Struct with unaligned fields

[Keith Thompson]

Perhaps I have lived a charmed life, then.
I hate to be that guy, but can I have a concrete example?

[...]

On a SPARC, this program:

#include <stdio.h>
int main(void)
{
struct foo {
char c;
int x;
} __attribute__((packed));
union u {
struct foo f;
long double align;
};
union u obj = { { 'a', 10 } };
int *ptr = &obj.f.x;
printf("obj.f.x = %d\n", obj.f.x);
fflush(stdout);
printf("*ptr = %d\n", *ptr);
return 0;
}

produces this output:

ptr = ffbff351
obj.f.x = 10
Bus error

(I needed the union to force the `struct foo` object to be word-aligned;
without it, the compiler places it at an odd address, causing the `int x`
member to be word-aligned.)

 

[Eric Sosman]

I am rather desperately trying to figure out why that's a problem.

Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
you try it, the program crashes with a bus error.
[...]

One RS/6000 system I recall did something even worse than
crashing: The hardware simply ignored the offending low-order
address bits, "rounding down" to the nearest aligned address.
Thus, it buzzed happily along (at full speed), having read or
written the wrong object ...

That was many years ago, and I don't know if the system's
descendants (many times renamed) still behave that way.

 

[James Harris]

....

It'll probably go down like a lead balloon here, but this is one case
where C++ will give you a much more elegant solution! You could abstract
all of the messy details behind what appears (to client code) to be a
conventional struct.

I'd like to find out some more about this. What would be a good search term?

James

 

[Les Cargill]

Perhaps I have lived a charmed life, then.
I hate to be that guy, but can I have a concrete example?

[...]

On a SPARC, this program:

#include <stdio.h>
int main(void)
{
struct foo {
char c;
int x;
} __attribute__((packed));
union u {
struct foo f;
long double align;
};
union u obj = { { 'a', 10 } };
int *ptr = &obj.f.x;
printf("obj.f.x = %d\n", obj.f.x);
fflush(stdout);
printf("*ptr = %d\n", *ptr);
return 0;
}

produces this output:

ptr = ffbff351
obj.f.x = 10
Bus error

(I needed the union to force the `struct foo` object to be word-aligned;
without it, the compiler places it at an odd address, causing the `int x`
member to be word-aligned.)

Thanks, Keith. Such sadistic hardware!

 

[Keith Thompson]

Stephen Sprunk wrote: [...]
Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
you try it, the program crashes with a bus error.

Perhaps I have lived a charmed life, then.
I hate to be that guy, but can I have a concrete example?

[...]

On a SPARC, this program:
[snip]

produces this output:

ptr = ffbff351
obj.f.x = 10
Bus error

(I needed the union to force the `struct foo` object to be word-aligned;
without it, the compiler places it at an odd address, causing the `int x`
member to be word-aligned.)

Thanks, Keith. Such sadistic hardware!

Sadistic? I don't think so.

My understanding is that supporting misaligned access requires
extra work in designing the CPU hardware. Not supporting it left
more resources available, both in design effort and chip area,
to support other features (lots of registers, for example) and
better performance.

And 90+% of the time, keeping operands properly aligned isn't much
of a problem anyway; the compiler takes care of it for you.

Perhaps the tradeoffs have changed since SPARC was designed, and
supporting unaligned access makes more sense now. But you can see
from my (snipped) example how convoluted your code has to be to
trigger a problem.

 

[Ian Collins]

You can crease classes similar to:

class nbou4 // Unsigned, four bytes
{
private:
unsigned char a, b, c, d;
public:
operator unsigned long()
{
return (unsigned long) ( ((unsigned long)a)<<24
| ((unsigned long)b)<<16
| ((unsigned long)c)<<8
| ((unsigned long)d)<<0
);
}
nbou4& operator=(unsigned long l)
{
a = (unsigned char)(((unsigned long)l)>>24)&0x000000ff;
b = (unsigned char)(((unsigned long)l)>>16)&0x000000ff;
c = (unsigned char)(((unsigned long)l)>>8)&0x000000ff;
d = (unsigned char)(((unsigned long)l)>>0)&0x000000ff;
return *this;
}
};

and then use them in a structure:

Yes, I use something similar (a generic template with fewer casts!)
except I use a pointer to the start byte rather than individual chars.

struct z
{
nbou4 fourbyteunsigned;
nbou2 twobyteunsigned;
};

And then use them as fairly ordinary longs in your code. IOW:

x = zinstance.fourbyteunsigned + 7;
zinstance.twobyteunsigned = atoi("37");

This depends on eight bits bytes, four byte longs, and that the
compiler not insert any alignment between the items, so that the
individual chars within the item are actually contiguous with the
preceding and following items.

Using a pointer to the start removes this minor risk.

 

[Tim Prince]

Stephen Sprunk wrote: [...]
Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
you try it, the program crashes with a bus error.

Perhaps I have lived a charmed life, then.
I hate to be that guy, but can I have a concrete example?
[...]

On a SPARC, this program:
[snip]

produces this output:

ptr = ffbff351
obj.f.x = 10
Bus error

(I needed the union to force the `struct foo` object to be word-aligned;
without it, the compiler places it at an odd address, causing the `int x`
member to be word-aligned.)

Thanks, Keith. Such sadistic hardware!

Sadistic? I don't think so.

My understanding is that supporting misaligned access requires
extra work in designing the CPU hardware. Not supporting it left
more resources available, both in design effort and chip area,
to support other features (lots of registers, for example) and
better performance.

And 90+% of the time, keeping operands properly aligned isn't much
of a problem anyway; the compiler takes care of it for you.

Perhaps the tradeoffs have changed since SPARC was designed, and
supporting unaligned access makes more sense now. But you can see
from my (snipped) example how convoluted your code has to be to
trigger a problem.

Several current compilers use split moves to handle what appear to be
mis-aligned access, in case the hardware support may be even worse than
using multiple instructions, even though neither choice will fail outright.
For example, initial implementations of AVX-256 perform better when
mis-aligned 256-bit moves are split into 128-bit move instructions. It's
impractical to sort out whether a newer hardware platform is running
which would perform better with a 256-bit unaligned move.
Current 512-bit wide architectures support mis-alignment only by the
compiler choosing split moves, at a performance penalty.

 

[Stephen Sprunk]

You can crease classes similar to:

class nbou4 // Unsigned, four bytes
{
private:
unsigned char a, b, c, d;
public:
operator unsigned long()
{
return (unsigned long) ( ((unsigned long)a)<<24
| ((unsigned long)b)<<16
| ((unsigned long)c)<<8
| ((unsigned long)d)<<0
);
}
nbou4& operator=(unsigned long l)
{
a = (unsigned char)(((unsigned long)l)>>24)&0x000000ff;
b = (unsigned char)(((unsigned long)l)>>16)&0x000000ff;
c = (unsigned char)(((unsigned long)l)>>8)&0x000000ff;
d = (unsigned char)(((unsigned long)l)>>0)&0x000000ff;
return *this;
}
};

Ah, I figured this had to be possible but didn't think of overloading
those particular operators; that's clever.

This depends on eight bits bytes, four byte longs,

Why not make that dependency explicit by using uint8_t and uint32_t
instead of unsigned char and unsigned long? That also has a side
benefit of dealing with machines that have a 64-bit unsigned long.

and that the compiler not insert any alignment between the items,
so that the individual chars within the item are actually contiguous
with the preceding and following items.

Certainly not 100% portable, but it wil work on many implementations.

Lack of padding is not guaranteed, true, but alignon(nbou4) is almost
certainly 1, so why would an implementation insert padding? I've never
heard of one that would.

S

 

[Les Cargill]

Stephen Sprunk wrote: [...]
Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
you try it, the program crashes with a bus error.

Perhaps I have lived a charmed life, then.
I hate to be that guy, but can I have a concrete example?
[...]

On a SPARC, this program:
[snip]

produces this output:

ptr = ffbff351
obj.f.x = 10
Bus error

(I needed the union to force the `struct foo` object to be word-aligned;
without it, the compiler places it at an odd address, causing the `int x`
member to be word-aligned.)

Thanks, Keith. Such sadistic hardware!

Sadistic? I don't think so.

I forgot to smiley. Here:

My understanding is that supporting misaligned access requires
extra work in designing the CPU hardware.

So they push this off to the toolchain. But the toolchain
doesn't support it...

Not supporting it left
more resources available, both in design effort and chip area,
to support other features (lots of registers, for example) and
better performance.

And 90+% of the time, keeping operands properly aligned isn't much
of a problem anyway; the compiler takes care of it for you.

I believe that was the original point - that GNU offers
"#prgama pack" and that I've see it work. Seems the best approach,
if it was available.

Perhaps the tradeoffs have changed since SPARC was designed, and
supporting unaligned access makes more sense now. But you can see
from my (snipped) example how convoluted your code has to be to
trigger a problem.

Certainly. In this case, it appears to be a vestige of formats
developed under 8-bit machines.

 

[Ian Collins]

Keith Thompson wrote:
Stephen Sprunk wrote: [...]
Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
you try it, the program crashes with a bus error.

Perhaps I have lived a charmed life, then.
I hate to be that guy, but can I have a concrete example?
[...]

On a SPARC, this program:
[snip]

produces this output:

ptr = ffbff351
obj.f.x = 10
Bus error

(I needed the union to force the `struct foo` object to be word-aligned;
without it, the compiler places it at an odd address, causing the `int x`
member to be word-aligned.)

Thanks, Keith. Such sadistic hardware!

Sadistic? I don't think so.

I forgot to smiley. Here:

My understanding is that supporting misaligned access requires
extra work in designing the CPU hardware.

So they push this off to the toolchain. But the toolchain
doesn't support it...

In the case of SPARC, yes it does, but at a significant performance
cost. Even with the appropriate hardware support, misaligned access is
still a performance killer.

 

[Keith Thompson]

Keith Thompson wrote:
Stephen Sprunk wrote: [...]
Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
you try it, the program crashes with a bus error.

Perhaps I have lived a charmed life, then.
I hate to be that guy, but can I have a concrete example?
[...]

On a SPARC, this program:
[snip]

produces this output:

ptr = ffbff351
obj.f.x = 10
Bus error

(I needed the union to force the `struct foo` object to be word-aligned;
without it, the compiler places it at an odd address, causing the `int x`
member to be word-aligned.)

Thanks, Keith. Such sadistic hardware!

Sadistic? I don't think so.

I forgot to smiley. Here:

My understanding is that supporting misaligned access requires
extra work in designing the CPU hardware.

So they push this off to the toolchain. But the toolchain
doesn't support it...

It does, mostly; gcc supports #pragma pack. (The native C compiler
also supports #pragma pack. Oddly, my test program doesn't crash
when compiled with Sun C 5.9; I haven't yet figured out why.)
As long as you refer to an misaligned member by name, the compiler
will do whatever is necessary to access it correctly. It only
causes problems if you go behind the compiler's back by grabbing
a pointer to a misaligned member.

One solution might be to create a compiler-specific attribute for
pointer types, so that taking an `int*` address of a misaligned
member is illegal, but taking a `__misaligned int*` (or whatever
syntax you prefer) address is legal -- and dereferencing such a
pointer does some extra work.

Another point in favor of requiring strict alignment: If you do
it accidentally (say, by treating a slice of a char array as an
int), performance is likely to suffer, even on an x86. By making
misaligned access a fatal error, you can catch such problems sooner.
I'm not saying that's necessarily enough of a reason by itself to
require strict alignment.

[...]

 

[Tim Rentsch]

To illustrate, the FAT header starts like this:

8-byte BS_OEMName
2-byte BPB_BytsPerSec
1-byte BPB_SecPerClus
2-byte BPB_RsvdSecCnt
...

As you can see, it is impossible to align the last of those fields,
BPB_RsvdSecCnt. The same is true of some of the fields that follow.

I've seen the c faq at http://c-faq.com/struct/padding.html and know
that there are no ideal solutions. This post is to ask for
suggestions as to how to address this well and portably.

Anyone already addressed the same sort of issue?

[One] way might be to use a byte (ie. char) array, and to use
type-punning to extract the fields that are wider than one-byte.
[snip elaboration]

Taking this approach will almost certainly lead to undefined
behavior caused by not following effective type rules. It
is always allowed to access an object of any effective type
using an lvalue of character type, but not vice versa. Of
course it's possible there will be no adverse manifestation
of the UB, but it's not a good idea to put yourself in the
position of having to rely on that.

 

[Tim Rentsch]

[illustrating an approach to using an "in-place" struct]

/*
* Define the FAT geometry table at the start of a volume
*
* The numbers preceding each field give the byte offsets of that field.
* Fields which cannot be aligned are split into arrays of octets.
*/

struct bootsect {
/* 0 */ ui8 jump[3],
/* 3 */ ui8 oem[8],
/* 11 */ ui8 secsize[2], /* Split due to alignment */
/* 13 */ ui8 sec_per_clus,
/* 14 */ ui16 sec_rsvd,
/* 16 */ ui8 fats,
/* 17 */ ui8 rootents[2], /* Split due to alignment */
/* 19 */ ui8 totsec16[2], /* Split due to alignment */
/* 21 */ ui8 medium_type,
/* 22 */ ui16 fatsize16,
/* 24 */ ui16 sec_per_trk,
/* 26 */ ui16 heads,
/* 28 */ ui32 hid_sec,
/* 32 */ ui32 totsec32,
/* 36 */ ui8 drv_num,
/* 37 */ ui8 rsvd1,
/* 38 */ ui8 bootsig,
/* 39 */ ui8 volid[4], /* Split due to alignment */
/* 43 */ ui8 vollab[11],
/* 54 */ ui8 fstype[8]
/* 63 */
};
typedef struct bootsect bootsect_t;

As you can see, in this case and as long as I have got the layout
right, there should be only four fields which would have to be split
due to alignment issues. The other fields should be accessible
directly and so not need to be converted first.

If you do go down this path, be warned that accessing
the "split" fields using casted pointer values easily
may wander off into undefined behavior, even on machines
with no alignment issues, because of problems with
effective type rules. Basically, if you declare a field
to be of character type, it must be accessed using only
character lvalues (not counting assigning the whole struct
or something like that).

 

[Les Cargill]

Keith Thompson wrote:
Stephen Sprunk wrote:
[...]
Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
you try it, the program crashes with a bus error.

Perhaps I have lived a charmed life, then.
I hate to be that guy, but can I have a concrete example?
[...]

On a SPARC, this program:

[snip]

produces this output:

ptr = ffbff351
obj.f.x = 10
Bus error

(I needed the union to force the `struct foo` object to be word-aligned;
without it, the compiler places it at an odd address, causing the `int x`
member to be word-aligned.)

Thanks, Keith. Such sadistic hardware!

Sadistic? I don't think so.

I forgot to smiley. Here:

My understanding is that supporting misaligned access requires
extra work in designing the CPU hardware.

So they push this off to the toolchain. But the toolchain
doesn't support it...

It does, mostly; gcc supports #pragma pack. (The native C compiler
also supports #pragma pack. Oddly, my test program doesn't crash
when compiled with Sun C 5.9; I haven't yet figured out why.)
As long as you refer to an misaligned member by name, the compiler
will do whatever is necessary to access it correctly. It only
causes problems if you go behind the compiler's back by grabbing
a pointer to a misaligned member.

I am no longer certain why so much was said about
alignment issues, then.

One solution might be to create a compiler-specific attribute for
pointer types, so that taking an `int*` address of a misaligned
member is illegal, but taking a `__misaligned int*` (or whatever
syntax you prefer) address is legal -- and dereferencing such a
pointer does some extra work.

Not too bad an approach.

Another point in favor of requiring strict alignment: If you do
it accidentally (say, by treating a slice of a char array as an
int), performance is likely to suffer, even on an x86. By making
misaligned access a fatal error, you can catch such problems sooner.
I'm not saying that's necessarily enough of a reason by itself to
require strict alignment.

[...]

 

[Ian Collins]

I'm not sure how that helps with inter-item padding.

Because there is only one member (the pointer)!

 

[David Thompson]

On 24-Aug-13 14:26, Robert Wessel wrote:

Ah, I figured this had to be possible but didn't think of overloading
those particular operators; that's clever.


Why not make that dependency explicit by using uint8_t and uint32_t
instead of unsigned char and unsigned long? That also has a side
benefit of dealing with machines that have a 64-bit unsigned long.

Yes and yes.

Lack of padding is not guaranteed, true, but alignon(nbou4) is almost
certainly 1, so why would an implementation insert padding? I've never
heard of one that would.

The original 16-bit Tandem NonStop, and I believe its predecessor
HP3000, had different format pointers to byte (char) and everything
larger (word=2bytes or more). They used 'word' pointers for structs,
so all structs had to be word = 2-byte aligned. This only mattered for
struct-in-struct, because all top-level variables were word-aligned
anyway. When they extended to 32-bit NonStop II aka NS2 they added
32-bit all-byte pointers, but included the original instructions and
pointers, retronymed NS1+ (the plus was some marketroid's invention,
nothing was actually added), so to allow pointer conversions to work
as long as you stayed in the 16-bit addressable area (essentially the
stack and "low" static and heap, the compiler default and thus an easy
rule to follow) structs still had to be 2-byte aligned.

"NS1+" and NS2 hardware is gone now, but TNS was mostly used for
enterprise-critical systems like airline reservations and bank ATMs.
The owners of these systems are reluctant and slow to make changes
whose failure might cost billions of dollars, so successor hardware
today -- commodity byte RISC -- can still run 30-year-old NS2 and even
NS1 code, nearly matching IBM's better-known S/360 clan.

 

[Stephen Sprunk]

Lack of padding is not guaranteed, true, but alignon(nbou4) is
almost certainly 1, so why would an implementation insert padding?
I've never heard of one that would.

The original 16-bit Tandem NonStop, and I believe its predecessor
HP3000, had different format pointers to byte (char) and everything
larger (word=2bytes or more). They used 'word' pointers for structs,
so all structs had to be word = 2-byte aligned. This only mattered
for struct-in-struct, because all top-level variables were
word-aligned anyway. [snip later history]

Ah; that's interesting. Does the same hold for other word-addressed
systems, or is it peculiar to the above example?

S

 

[Stephen Sprunk]

On Sat, 24 Aug 2013 16:16:08 -0500, Stephen Sprunk
Lack of padding is not guaranteed, true, but alignon(nbou4) is
almost certainly 1, so why would an implementation insert
padding? I've never heard of one that would.

The original 16-bit Tandem NonStop, and I believe its predecessor
HP3000, had different format pointers to byte (char) and
everything larger (word=2bytes or more). They used 'word'
pointers for structs, so all structs had to be word = 2-byte
aligned. This only mattered for struct-in-struct, because all
top-level variables were word-aligned anyway. [snip later
history]

Ah; that's interesting. Does the same hold for other
word-addressed systems, or is it peculiar to the above example?

Some word addressed machines used the same size pointer for chars,
but with a different internal format. There were examples of both
char pointers having the character offset stuffed in at both ends of
the address. ...

I was aware of special char-offset pointers; I just hadn't heard of
structs (even ones with only char members) having to be aligned on a
word boundary.

S

 

[David Thompson]

26, Robert Wessel wrote:
and that the compiler not insert any alignment between the
items, so that the individual chars within the item are actually
contiguous with the preceding and following items.

Certainly not 100% portable, but it wil work on many
implementations.

Lack of padding is not guaranteed, true, but alignon(nbou4) is
almost certainly 1, so why would an implementation insert padding?
I've never heard of one that would.

The original 16-bit Tandem NonStop, and I believe its predecessor
HP3000, had different format pointers to byte (char) and everything
larger (word=2bytes or more). They used 'word' pointers for structs,
so all structs had to be word = 2-byte aligned. This only mattered
for struct-in-struct, because all top-level variables were
word-aligned anyway. [snip later history]

To be clear re the below: TNS1 pointers were the same *size*, 16 bits.
Byte pointer was high 15 bits select word and low bit select byte,
word pointer was plain 16 bits, but the compilers+linker put variables
and the hardware put the (upward!) stack in 00000-77777; 100000-177777
was used by the language runtimes (things like FILE blocks in C) and
if needed (rarely) could also be manually allocated/managed.

Some word addressed machines used the same size pointer for chars, but
with a different internal format. There were examples of both char
pointers having the character offset stuffed in at both ends of the
address. IOW, both (8*WordAddress+CharOffset) and
(WordAddress+(CharOffset<<29)) happened (obviously assuming eight byte
words for this example), and the compiler would generate the code to
deal with that as necessary.

PDP-10 (and PDP-6) had an interesting variant on this. It had 36-bit
word and most instructions accessed a word using a 18-bit address
(optionally with index/offset and indirect); one special group of
instructions accessed the left or right half-word (selected by the
instruction, at compile time); and one special group of instructions
could access *any* bit-field from 1 bit up to the whole word, using a
'byte pointer' with the word address in the right half like a normal
pointer, and the bit offset and bit width in the left half. Since C
requires that chars be at least 8 bits, fixed, and a u-char array
access all bits of memory, it could only use 9, 12, or 18 bit bytes.
AFAIK there was no C for PDP-10 back in the 1970s (when C and even
Unix were mostly viewed as a curious experiment); there was talk a few
years ago in alt.sys.pdp10 about some people doing a gcc target (there
are still a few hardware machines going, and also good simulators) and
I'm pretty sure they used 9.