Ranting about JVM's default memory limits...

[Lew]

(Please interleave whom you're quoting)

I don't think the limit was ever implemented as a desired feature. It is
the oldest java bug resulting from GC's implementation.

What bug is that, how did it result from GC's peanut butter, and how is that
influencing the -Xmx festival?

Are you referring to an ethnic limit, or just that there is one?

--
Lew


- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[Freemasonry, occult, Kabbalah, deception,
Illuminati, NWO]

Commentator:

"It has been described as "the biggest, richest, most secret
and most powerful private force in the world"... and certainly,
"the most deceptive", both for the general public, and for the
first 3 degrees of "initiates": Entered Apprentice, Fellow Craft,
and Master Mason (the basic "Blue Lodge")...

These Initiates are purposely deceived!, in believing they know
every thing, while they don't know anything about the true Masonry...
in the words of Albert Pike, whose book "Morals and Dogma"
is the standard monitor of Masonry, and copies are often
presented to the members"

Albert Pike:

"The Blue Degrees [first three degrees in freemasonry]
are but the outer court of the Temple.
Part of the symbols are displayed there to the Initiate, but he
is intentionally mislead by false interpretations.

It is not intended that he shall understand them; but it is
intended that he shall imagine he understand them...
but it is intended that he shall imagine he understands them.
Their true explication is reserved for the Adepts, the Princes
of Masonry.

...it is well enough for the mass of those called Masons
to imagine that all is contained in the Blue Degrees;
and whoso attempts to undeceive them will labor in vain."

--- Albert Pike, Grand Commander, Sovereign Pontiff
of Universal Freemasonry,
Morals and Dogma", p.819.

[Pike, the founder of KKK, was the leader of the U.S.
Scottish Rite Masonry (who was called the
"Sovereign Pontiff of Universal Freemasonry,"
the "Prophet of Freemasonry" and the
"greatest Freemason of the nineteenth century."),
and one of the "high priests" of freemasonry.

He became a Convicted War Criminal in a
War Crimes Trial held after the Civil Wars end.
Pike was found guilty of treason and jailed.
He had fled to British Territory in Canada.

Pike only returned to the U.S. after his hand picked
Scottish Rite Succsessor James Richardon 33° got a pardon
for him after making President Andrew Johnson a 33°
Scottish Rite Mason in a ceremony held inside the
White House itself!]

 

[Mark Space]

OK, I'd like to blow off a little steam about the way Sun's JVM handles
memory by default.

Basically, I don't see any option for "grow memory until the OS runs
out" which the default behavior for every other desktop application that
I personally use.

It's completely impractical to ask the user to set this. And grossly
inconvenient to ask the developer to do it for each and every user,
separately. On my machine, if I guess a max heap size (-Xmn) too small
programs run out of memory for no reason. If I guess a max heap size to
large, then the JVM refuses to start because it can grab all the memory
requested on that system.

Java is the only language that exhibits this highly undesirable feature.
Every other app on my desktop (eg., Firefox or Thunderbird) runs
happily until the OS tells it "You're out of memory" then it takes some
kind of action. This allows the user (and the programmer) to take full
advantage of the installed memory with out having to try to predict
future usage patterns.

Gah, I'm really fuming about this. No wonder no one writes desktop apps
in Java, it's grossly impractical.

In a server environment, I can see that one really wants to limit memory
usage. It's a much more controlled situation, and professionally
managed (hopefully, relatively speaking).

For clients, I can see the need to occasionally specify memory
constraints. But I'd really really like to see Sun make the default to
just use memory until the OS says "No more" and then deal with the
results appropriately.


Any further comments? Have I missed the mother of all memory options
for Sun's JVM somewhere? I'd be happy if I had.

Thanks for listening....

 

[Arne Vajhøj]

OK, I'd like to blow off a little steam about the way Sun's JVM handles
memory by default.

Basically, I don't see any option for "grow memory until the OS runs
out" which the default behavior for every other desktop application that
I personally use.

It's completely impractical to ask the user to set this. And grossly
inconvenient to ask the developer to do it for each and every user,
separately. On my machine, if I guess a max heap size (-Xmn) too small
programs run out of memory for no reason. If I guess a max heap size to
large, then the JVM refuses to start because it can grab all the memory
requested on that system.

Java is the only language that exhibits this highly undesirable feature.
Every other app on my desktop (eg., Firefox or Thunderbird) runs
happily until the OS tells it "You're out of memory" then it takes some
kind of action. This allows the user (and the programmer) to take full
advantage of the installed memory with out having to try to predict
future usage patterns.

Gah, I'm really fuming about this. No wonder no one writes desktop apps
in Java, it's grossly impractical.

In a server environment, I can see that one really wants to limit memory
usage. It's a much more controlled situation, and professionally
managed (hopefully, relatively speaking).

For clients, I can see the need to occasionally specify memory
constraints. But I'd really really like to see Sun make the default to
just use memory until the OS says "No more" and then deal with the
results appropriately.

Any further comments? Have I missed the mother of all memory options
for Sun's JVM somewhere? I'd be happy if I had.

Actually I like the "you have to specify a limit and if you exceed that
then you have a problem because apparantly you do not know what your
program are doing" philosophy.

But:
- it is unusual compared to anything else which makes it a problem
- the current default is from a previous decade

I guess I could live with 256m or 512m as default.

Arne

 

[Mark Space]

Actually I like the "you have to specify a limit and if you exceed that
then you have a problem because apparantly you do not know what your
program are doing" philosophy.

I can manage it fine on my own machine, but it really seems like it
would be a huge problem to manage on any kind of decently sized
installed base.

I just can't figure out how I would manage some Java Web Start app that
I wrote, for example, other than to greatly inconvenience the user and
make them do it.

But:
- it is unusual compared to anything else which makes it a problem

Which makes users complain. And defeats the purpose of having an OS
with memory management if I have to manage it myself anyway. What
happens when someone installs more memory? I have to set the limit
higher on each app to take advantage of it? Agh! Why?

(That's a rhetorical "why?" I know you don't make policy for Sun.)

- the current default is from a previous decade

My point is the whole idea of a fixed limit is, like, 3 decades old.
Hello, 1978 called, and they want their (lack of) memory management back.

I guess I could live with 256m or 512m as default.

I'd like to see Sun implement some sort of adaptive algorithm that
increased the max heap in response to GC pressure, and was able to
reduce the max heap (return memory to the OS) if the GC was able to free
a larger percentage of memory in any pass.


Anyhoo, again thanks for letting me vent....

 

[Arne Vajhøj]

Which makes users complain. And defeats the purpose of having an OS
with memory management if I have to manage it myself anyway. What
happens when someone installs more memory? I have to set the limit
higher on each app to take advantage of it?

No.

Xmx is virtual memory not RAM.

So you may change it when moving from 32 bit to 64 bit OS.

It does not matter if you have 256 MB or 4 GB of RAM.

My point is the whole idea of a fixed limit is, like, 3 decades old.
Hello, 1978 called, and they want their (lack of) memory management back.

You have GC and virtual memory. It can not be more managed.

I don't think limits are oldfashioned.

I'd like to see Sun implement some sort of adaptive algorithm that
increased the max heap in response to GC pressure, and was able to
reduce the max heap (return memory to the OS) if the GC was able to free
a larger percentage of memory in any pass.

I belive that it already does. From Xms up to Xmx.

Arne

 

[Mark Space]

No.

Xmx is virtual memory not RAM.

Virtual memory in general tracks physical RAM (2 to 4 times physical RAM
is a good rule of thumb), but it doesn't matter. Increase virtual
memory on a system, you still have to manually change it on each Java app.

I don't think limits are oldfashioned.

I guess I just 1000% disagree here. Inappropriate for client machines,
at least.

I belive that it already does. From Xms up to Xmx.

I need the JVM to supply Xmx=available system memory for me. That's the
point.

 

[Arne Vajhøj]

Virtual memory in general tracks physical RAM (2 to 4 times physical RAM
is a good rule of thumb),

No.

Virtual memory on all 32 bit systems is 4 GB (on Windows 32 bit
without /3GB only 2 GB is available for apps).

You are thinking about the MS usage of the term "virtual memory" - for
reasons unknown to me they use it about page file.

Xmx and RAM has very little to do with each other.

You can use -Xmx1600m on a system with 256 MB of RAM. I may
run a bit slow if you actually use that much mem though !

> but it doesn't matter. Increase virtual
memory on a system, you still have to manually change it on each Java app.

Xmx is only related to the 32/64 bitness of the OS not to amount of
RAM or size of pagefile.

With the two exceptions that too little RAM makes the program slow
and systems that run out of page file crashes.

If you upgrade from 1 GB RAM + 2 GB page file to
2 GB RAM + 4 GB page file then you do not need
to change your Xmx - you can use and you can not
get higher than approx. -Xmx1700m on both
configs.

I need the JVM to supply Xmx=available system memory for me. That's the
point.

That is a valid wish. You can always send it to SUN.

Arne

 

[Mark Space]

Virtual memory on all 32 bit systems is 4 GB (on Windows 32 bit
without /3GB only 2 GB is available for apps).

Ah, good point. There are of course physical limits of virtual memory.

You are thinking about the MS usage of the term "virtual memory" - for
reasons unknown to me they use it about page file.

Xmx and RAM has very little to do with each other.

You can use -Xmx1600m on a system with 256 MB of RAM. I may
run a bit slow if you actually use that much mem though !

Not me. I can't set -Xmx2G on my system (32 bit Windows, 64 bit Intel).
So I have to be aware of the limits on each separate platform.

And still why should I be required to upgrade all of my programs just
because (suppose) MS updates their OS to support 4G more virtual memory?
It's silly. The JVM should just use what's available.

(Back in the old days, Intel supported more virtual memory than address
lines by adding a few more "states" to their external bus: S0-S2. Dunno
if that's still available.)

Xmx is only related to the 32/64 bitness of the OS not to amount of
RAM or size of pagefile.

I think there are some interactions which vary from system to system.
See above for 2G not being accepted on my machine. "Write once, test
everywhere." It seems to be rearing it's ugly head for me.

If you upgrade from 1 GB RAM + 2 GB page file to
2 GB RAM + 4 GB page file then you do not need
to change your Xmx - you can use and you can not
get higher than approx. -Xmx1700m on both
configs.

Well it seems to me that 1700M is just a temporary, platform dependent
limitation. What happens when hardware and OS upgrades? There's a lot
of potential MANIFEST files out there, already installed, that have to
be updated. Why push that work onto developers?

That is a valid wish. You can always send it to SUN.

I was just making sure it was valid before sending Sun an official
gripe-o-gram.

 

[jolz]

In a server environment, I can see that one really wants to limit memory

usage. It's a much more controlled situation, and professionally
managed (hopefully, relatively speaking).

I don't think the limit was ever implemented as a desired feature. It is
the oldest java bug resulting from GC's implementation.

 

[Arne Vajhøj]

Not me. I can't set -Xmx2G on my system (32 bit Windows, 64 bit Intel).

No. Not 2g but about 1700m - that is the limitation of
32 bit Windows.

So I have to be aware of the limits on each separate platform.

If you use 1600m on 32 bit OS'es and 16g on 64 bit OS'es I think
you will be covered.

And still why should I be required to upgrade all of my programs just
because (suppose) MS updates their OS to support 4G more virtual memory?

MS can not update 32 bit Windows to support that, It would not be 32 bit
anymore.

I think there are some interactions which vary from system to system.
See above for 2G not being accepted on my machine.

Many 32 bit OS including Windows split the total of 4 GB in system
space and process space.

Java can only use process space.

2 GB - "some stuff" is what is available.

Well it seems to me that 1700M is just a temporary, platform dependent
limitation. What happens when hardware and OS upgrades?

Hardware: nothing.

Upgrade to another 32 bit OS: nothing.

Upgrade to a 64 bit OS: endless possibilities !

Arne

 

[jolz]

What bug is that,

The bug is called OutOfMemoryError. Developer has no obvious way of
specifying maximum available memory and has to set lower limit then
possible. If it results in OutOfMemoryError it is jvm's bug.

how did it result from GC's peanut butter, and how is that
influencing the -Xmx festival?

Heap size affects GC's performace. Without java's own memory menagement
there would be no reason to enforce any limits. It's operating system's job.

 

[Mark Space]

I don't think the limit was ever implemented as a desired feature. It is
the oldest java bug resulting from GC's implementation.

I don't think "bug" is fair -- it's working as intended, and for server
environments it's probably a good idea.

For clients, "undesired feature" might be a good description.
"Boneheaded move" might be a better one.

 

[Lew]

The bug is called OutOfMemoryError. Developer has no obvious way of
specifying maximum available memory and has to set lower limit then
possible. If it results in OutOfMemoryError it is jvm's bug.

No, that sneakily means the maintainer's bug.

Please do not quote the poster. You should killfile him.

--
Lew


- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
"But it's not just the ratty part of town," says Nixon.
"The upper class in San Francisco is that way.

The Bohemian Grove (an elite, secrecy-filled gathering outside
San Francisco), which I attend from time to time.

It is the most faggy goddamned thing you could ever imagine,
with that San Francisco crowd. I can't shake hands with anybody
from San Francisco."

Chicago Tribune - November 7, 1999
NIXON ON TAPE EXPOUNDS ON WELFARE AND HOMOSEXUALITY
by James Warren
http://econ161.berkeley.edu/Politics/Nixon_on_Tape.html

The Bohemian Grove is a 2700 acre redwood forest,
located in Monte Rio, CA.
It contains accommodation for 2000 people to "camp"
in luxury. It is owned by the Bohemian Club.

SEMINAR TOPICS Major issues on the world scene, "opportunities"
upcoming, presentations by the most influential members of
government, the presidents, the supreme court justices, the
congressmen, an other top brass worldwide, regarding the
newly developed strategies and world events to unfold in the
nearest future.

Basically, all major world events including the issues of Iraq,
the Middle East, "New World Order", "War on terrorism",
world energy supply, "revolution" in military technology,
and, basically, all the world events as they unfold right now,
were already presented YEARS ahead of events.

July 11, 1997 Speaker: Ambassador James Woolsey
former CIA Director.

"Rogues, Terrorists and Two Weimars Redux:
National Security in the Next Century"

July 25, 1997 Speaker: Antonin Scalia, Justice
Supreme Court

July 26, 1997 Speaker: Donald Rumsfeld

Some talks in 1991, the time of NWO proclamation
by Bush:

Elliot Richardson, Nixon & Reagan Administrations
Subject: "Defining a New World Order"

John Lehman, Secretary of the Navy,
Reagan Administration
Subject: "Smart Weapons"

So, this "terrorism" thing was already being planned
back in at least 1997 in the Illuminati and Freemason
circles in their Bohemian Grove estate.

"The CIA owns everyone of any significance in the major media."

--- Former CIA Director William Colby

When asked in a 1976 interview whether the CIA had ever told its
media agents what to write, William Colby replied,
"Oh, sure, all the time."

[More recently, Admiral Borda and William Colby were also
killed because they were either unwilling to go along with
the conspiracy to destroy America, weren't cooperating in some
capacity, or were attempting to expose/ thwart the takeover
agenda.]

 

[Tom Anderson]

I don't think "bug" is fair -- it's working as intended, and for server
environments it's probably a good idea.

For clients, "undesired feature" might be a good description.
"Boneheaded move" might be a better one.

I mostly agree with you - it's annoying to have to set maximum heap size
explicitly.

The problem is, in a garbage collected language, how do you set it
automatically?

In a manual memory management system, it's easy: if you run out of free
memory, you request more from the OS, because that's the only thing you
can do.

In a GC system, though, you always have the option of doing a collection.
So, if you run out of memory, do you grow the heap or collect? How do you
decide? You could say that you always collect, and only grow the heap if
that doesn't free enough memory, but that means your app will always be
running with nearly-full memory, which means it'll spend a lot of time
doing GC, which is not so hot. You could instead have a rule about having
a certain amount of free memory - say, if memory is more than a certain
percentage full after a full GC, grow the heap. But then you just replace
one parameter with another! I guess at least it's a parameter which is
more likely to have a sensible default that will work for all apps and all
points in history, which maximum heap size doesn't.

I'm not saying it can't be done, just that it's not trivial.

And i suppose this is really about the decision about when to grow the
heap, which java already has to make, not about maximum heap size.

tom

 

[Arne Vajhøj]

The problem is, in a garbage collected language, how do you set it
automatically?

In a manual memory management system, it's easy: if you run out of free
memory, you request more from the OS, because that's the only thing you
can do.

In a GC system, though, you always have the option of doing a
collection. So, if you run out of memory, do you grow the heap or
collect? How do you decide? You could say that you always collect, and
only grow the heap if that doesn't free enough memory, but that means
your app will always be running with nearly-full memory, which means
it'll spend a lot of time doing GC, which is not so hot. You could
instead have a rule about having a certain amount of free memory - say,
if memory is more than a certain percentage full after a full GC, grow
the heap. But then you just replace one parameter with another! I guess
at least it's a parameter which is more likely to have a sensible
default that will work for all apps and all points in history, which
maximum heap size doesn't.

I'm not saying it can't be done, just that it's not trivial.

And i suppose this is really about the decision about when to grow the
heap, which java already has to make, not about maximum heap size.

Exactly.

I am pretty sure that Java today runs GC before reaching Xmx.

Arne

 

[Stefan Ram]

percentage full after a full GC, grow the heap.

You have given a reasonable answer to your question.

But then you just replace one parameter with another!

But I do not understand this objection.

»You are always so hungry.«

(Problem.)

»Well I could double your daily food rations.«

(The correct solution!)

»But then I would just replace one parameter with another!«

(What?)

 

[Arne Vajhøj]

I find this interesting: could you provide some further information on
this?

It is pretty simple.

Windows divide the 4 GB address space in 2 parts:
2 GB for processes
2 GB for system

The 2 GB process space is per process. The 2 GB system space is shared
among processes.

By using the /3GB switch you can change that from 2+2 til 3+1.

The "some stuff" is all the non Java stuff that a process has
mapped.

I had previously been under the impression that the 2 GB limit comes
into effect for applications that are likely to do signed comparisons on
pointers - such as needing to call a legacy routine which returns
sub-zero values when it needs to indicate an error and an error-code in
one result.

No. Signed/unsigned of pointers has nothing to do with that.

To make a process use more than 2 GB on Windows, the executable image
has to be compiled with the /LARGEADDRESSAWARE compiler option (assuming
MSVC) : http://msdn.microsoft.com/en-us/library/wz223b1z.aspx

That is an option that tells Windows that the EXE can use
more than 2 GB.

It is used for:
- 32 bit windows with /3GB where the app can get 3 GB
- 64 bit windows where a 32 bit app can get 4 GB
(because the system stuff is elsewhere)

If I examing the file headers for a JDK, here's
jdk1.6.0_06\jre\bin\server\jvm.dll
Dumper Version 5.12.8078
Copyright (C) Microsoft Corp 1992-1998. All rights reserved.

Dump of file jvm.dll

PE signature found

File Type: DLL

FILE HEADER VALUES
14C machine (i386)
5 number of sections
47E8EFDF time date stamp Tue Mar 25 08:28:15 2008
0 file pointer to symbol table
0 number of symbols
E0 size of optional header
210E characteristics
Executable
Line numbers stripped
Symbols stripped
32 bit word machine
DLL

And for jdk1.6.0_06\bin\javaw.exe

Dump of file javaw.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
14C machine (i386)
4 number of sections
47E8C5CB time date stamp Tue Mar 25 05:28:43 2008
0 file pointer to symbol table
0 number of symbols
E0 size of optional header
10F characteristics
Relocations stripped
Executable
Line numbers stripped
Symbols stripped
32 bit word machine

Had those images been compiled with /LARGEADDRESSAWARE, the
"characteristics" section will include the message:

Application can handle large (>2GB) addresses

Without that flag being set in the executable files, I believe the OS
limits the memory the process may request. In theory, you can use
"editbin" to manually set it - but flipping bits in compiled exe files
seems to be asking for trouble.

I am surprised that those flags aren't set when Sun does a java build -
it would be nice if they were.

It is not as simple as that.

Most JVM's requires that the heap is continuous.

Even with /3GB Windows puts something in at the 2 GB mark, so
that most JVM's can not use the 3 GB.

The only JVM I know that can use 3 GB is BEA's JRockit.

Arne

 

[Mark Space]

I'm not saying it can't be done, just that it's not trivial.

Oh, I'm sure it's non trivial. But nothing worthwhile is trivial.

And i suppose this is really about the decision about when to grow the
heap, which java already has to make, not about maximum heap size.

Well, it's about the maximum heap size in the sense that I'm forced to
deal with the max heap size, and I'd strongly prefer not to. I'd much
rather leave the max heap size up to a JVM, which has system information
that a Java app is not privy to. (And the author isn't privy to, either.)

As for when to grow it, I guess you start with "when the smaller heap
fills up" and go from there.

Just to try to be forward looking, I'm sure -Xmx (fixed, absolute
maximum heap size) can stay for those who want to use it. But I'd also
like to see something like -Xmx:reluctant, normal, permissive where
those argument describe, in general terms, how often the GC decides to
grow the heap vs. run another object collection pass.

And I'd also like to see Java shrink the heap if it can. Return memory
to the OS please.

Yes, non-trivial to be sure, but greatly desired also.

 

[Arne Vajhøj]

I don't know if there's a
Windows Java run-time that's 64-bits, but there ought to be.

There are.

SUN, BEA, IBM and Oracle combined have for the OS's
Windows, Linux, Solaris and AIX. And the vendor specific
ones: HP for HP-UX and OpenVMS etc. I would even expect
Apples on MacOS X to support 64 bit, but if I remember
correctly then you can check that !

Arne

 

[Mark Space]

No. Not 2g but about 1700m - that is the limitation of
32 bit Windows.


If you use 1600m on 32 bit OS'es and 16g on 64 bit OS'es I think
you will be covered.

By the way, thanks very much for this information. Knowing where the
practical upper limit is for most systems is very handy.