malloc

[glen herrmannsfeldt]

This has been done in hardware as well.
http://en.wikipedia.org/wiki/Bounds_checking mentions VAX, B6500 and
Burroughs. By some reason this approach has not really catched on.

As I remember VAX allows pages to be marked read-only.
I remember Fortran compilers that would put constants in
read-only pages, such that they couldn't accidentally be
modified. With call-by-reference, modification of constants
has a long history in Fortran.

This is exactly what Electric Fence does. It is also slow as hell,
I guess the most slowdown comes from where the array ends in
a middle of a virtual memory page and efence

(snip)

In the 16 bit OS/2 days, when most people were still using 16-bit
DOS, I would allocate arrays by directly calling the OS/2
segment allocation routine, for a segment exactly the size
required. That length then goes into the hardware segment
descriptor register, such that hardware tests the bounds,
even on fetch. That was before paging, though.
(Eventually you run into the 8K segment descriptor per
process limit, but usually it works pretty well.)

If Intel had added a segment descriptor cache to any of
the processors, it would have greatly speeded up some of the
things that were done in the 16 bit, and early 32 bit, OS/2 days.

If done right, it would have delayed the need for 64 bit
processors by some years.

-- glen

 

[Goran]

     Yeah, it was probably my code.  Awfully nice of DEC to fix it
for me by patching VMS' C library, don't you think?

If you do have a legit library bug, you can just send a link to the
bug report.

I am not a mind reader, and in vast majority of cases, when people say
there's a library bug, there isn't.

Goran.

 

[Keith Thompson]

If you do have a legit library bug, you can just send a link to the
bug report.

I am not a mind reader, and in vast majority of cases, when people say
there's a library bug, there isn't.

I don't believe Eric is obligated to convince you that it really happened.

 

[Joshua Maurice]

Can you give me some more rational into this perhaps please? Or point
me in a right direction? Coding and supporting overcommit for programs
that use sparse arrays just seems like .. a braindead idea. Kernel
support that will result in random programs dying for a question use
case when equivalent or better alternatives likely exist in userspace?

Be careful not to conflate the linux OOM killer with overcommit failures.

One operating system (only noted as XXX in my notes from 26 Feb 1993) also
provides a SIGDANGER signal that can be caught when system memory is
low.

 Here is portions of the Standard Interpretation Request sent to X/Open: [notes]
===== End Notes =====

Oddly, I can't find any email regarding the resolution of this interpretation
request (my written notes from the meeting are in a box in the attic) and
there is nothing in the rationale here that would shed any light on it:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/malloc.html

scott

The justification given here is again sparse vectors or sparse
matrices. I'll repeat my earlier objection that just seems like an
entirely braindead thing to add to the kernel when there are easy
userspace workarounds. Hell, just increase swap by a large margin, and
with any sane modern virtual memory implementation, you'll get almost
the same result (except with a larger swap requirement, but not more
actual disk usage). So, the argument was that we want to allow this
buggy behavior to allow lazy userspace coders to run programs with
small swap? I was looking for specific case examples, something, that
goes into this use case besides another pithy one liner "it exists".
From google, the only references to virtual memory overcommit I can
seem to find are in relation to VMs, and the Linux feature. It's hard
for me to comment when I'm so relatively ignorant.

And yes, overcommit is separate from the Linux OOM killer. Still, your
options are rather limited as to what to do when a process writes to a
new page and you have no space left, and none of them are what I would
call "good".

 

[BGB]

Integrating Valgrind into the runtime isn't really practical. Valgrind is a
virtual machine. It'd be like integrating QEMU into the runtime.

not literally integrating Valgrind, but more as a debug option for every
time an assignment takes place through a raw pointer (serving a similar
role), the runtime will validate that it was a valid write.

it would be slow, but could be sane as a debug feature.

But as for Valgrind being too slow and only useful for debugging, I just
can't agree. Valgrind is excellent for _detecting_ bugs, not just diagnosing
them. If more people ran their regression tests on top of Valgrind (i.e.
`valgrind foo` instead of `foo') then the world of software would be a far
better place. I mean no offense, but frankly it boggles my mind how people
excuse themselves from running Valgrind, instead spending countless hours
tinkering with virtual memory paging tricks. (To be fair, I see it all the
time at work, too.)

my main issue with it is that it isn't available for Windows AFAIK.

Granted, when compared to C running C++ apps with Valgrind is slower still.
But that's because C++ apps are less judicious about object manipulation and
tend to use dynamic memory for every teeny-tiny allocation. All the more
reason to spot bugs earlier rather than later, as dynamic memory bugs can be
more maddening than simple stack overflows.

possibly.







Valgrind solves all of these issues by using a virtual machine. Then it's
relatively trivial to interpose CPU instructions to, e.g., introduce write
barriers. You can also instrument your code to communicate with Valgrind
during runtime.

yeah.

a downside here though is being primarily a Windows developer.

 

[Miles Bader]

yeah.

a downside here though is being primarily a Windows developer.

Would the work to port valgrind to windows be more than the work to
add some vaguely similar functionality to the compiler though?

-miles

 

[BGB]

not literally integrating Valgrind, but more as a debug option for every
time an assignment takes place through a raw pointer (serving a similar
role), the runtime will validate that it was a valid write.

it would be slow, but could be sane as a debug feature.

Valgrind maintains a bitmap of every byte of allocated memory. It can verify
every read and write operation at the [virtual] hardware level, including
whether a byte has been read from before being written to.

Although there's a tradeoff when filtering out the tremendous amount of
noise--usually involving stack or syscall operations--wherein occassionaly
some bugs are let through without a warning. This is almost never an issue
when dealing with dynamic memory from malloc() or new.

But in order to do all of these things it needs to watch everything from the
very beginning. You can't attach Valgrind to a running app; rather, you use
it to execute the application from the start. Instead of executing
`foo -bar', you execute `valgrind foo -bar'. In that way it's trivial to
integrate Valgrind into an IDE.

I would have assumed coarser checking, maybe in 16-byte units (or, at
least that would be convenient for me, as my memory manager allocates
memory in terms of 16-byte cells). actually, for smaller objects, the MM
allocates via a bitmap as well, but uses free-list allocation for larger
objects (once one gets into the kB range).

actually, the bulk of memory allocation in my apps tends to be objects
under 1kB (and many tiny objects exists).

That is, admittedly, a pretty decent excuse. Although, I primarily target
OpenBSD, which Valgrind has yet to support. This is one of those contexts
where code portability is priceless, apropos comp.lang.c and comp.lang.c++!
The real barrier, IME, isn't the code per se, but the build environment. But
writing portable code helps there as well, with less reliance on compiler
and build environment magic.

well, my stuff does build on Linux, but given it uses OpenGL, only
performs at usable speeds if the computer is actually booted from Linux
(rather than run from VMware).

 

[Eric Sosman]

If you do have a legit library bug, you can just send a link to the
bug report.

The bug and fix dated from 1988, if memory serves, or possibly
1989. VAX VMS (not OpenVMS), probably V5 but possibly V4, on a
MicroVAX II. The compiler was VAXC (a pre-ANSI implementation;
sorry I cannot recall the version number), and I think the run-time
library was VAXC$LIB (that's where the bug lay and was patched).

The company that issued the bug and then the patch went out
of business in 1998.

The company that bought the remains of the company also went
out of business, in 2002.

The company that bought the remains of the company that bought
the remains is still around, doing business under the name "Hewlett-
Packard Company." If you are interested, I suggest you contact them
about the possibility of making available their archives of dead
acquisitions' long-dead fix lists. (As I'm sure you're aware, putting
bug reports and fix lists on the Web was fairly unusual in 1988, since
Tim Berners-Lee's first Web browser was two years in the future.)

I am not a mind reader, and in vast majority of cases, when people say
there's a library bug, there isn't.

Agreed, on both points. For a number of reasons, libraries tend
to be less buggy than the calling code -- but "less buggy" does not
mean "bug-free." And to the other point, I'm convinced not only of
your inability to read minds, but of your difficulty in comprehending
text.

 

[Asger Joergensen]

Hi BGB

which are annoyingly difficult to track down sometimes...


it would be nice if compilers would offer an option (as a debugging feature) to
optionally put in bounds checking for many operations (it is also possible to do
this without adding fat pointers or fundamentally changing how the language
works, although it would still introduce a run-time cost).

Such a tool have been part of Borland/CodeGear/Embarcadero compilers for years
it is a debug option called CodeGuard, it also checks if You remember to delete
what You have created with new.
Just for debugging though, it is to slow for release code.

Best regards
Asger-P

 

[BGB]

Hi BGB


Such a tool have been part of Borland/CodeGear/Embarcadero compilers for years
it is a debug option called CodeGuard, it also checks if You remember to delete
what You have created with new.
Just for debugging though, it is to slow for release code.

yep.