Tuning suggestions, please?

[Twisted]

Twisted wrote :





It is a Web application. When no hits are happening, it should be doing
nothing. And my code does no networking.

Mind you I do have several timers running, but then the memory use goes
from 30M to 74M over a 2 hour period, then drops back to 30M, which
seems kind of excessive for timers.

5 or 6 bytes per millisecond on average. One new object created each
millisecond on average is more than enough to cause what you observe.
If the timers have resolutions of a few ms, there are a few of them,
and they generate new Dates each time they update ... there you go.

 

[Bret Schuhmacher]

Thanks Wojtek, that's exactly what I saw. Maybe it's normal.

Rgds,

Bret

 

[Bjorn Borud]

[Wojtek <[email protected]>]
|
| I have a Web app where I capture the used memory every ten
| minutes. The result is displayed as a bar graph. Even when the app is
| idle, the used memory looks like a saw tooth:
|
| * * *
| ** ** **
| *** *** ***
| *************

provided that you have an adequate sampling frequency, which in this
case would be at least two samples or more in the time between minor
collections, a distinct sawtooth shape is usually what you want for
programs like web apps. the period of the sawtooth should be some
multiple of the request time for the great majority of requests.
(which should be simple unless you have obscene variance).

-Bjørn

 

[Bjorn Borud]

["Bret Schuhmacher" <[email protected]>]
|
| I've run visualgc and the GC time shows 94 collections, 59.81ms and unknown
| GC reason. The Eden space graph is up and down like a yo-yo very regularly.
| Is that normal? I don't know where the tenured GC is, though. I have an
| image of the tool running, but probably can't post it to this non binary
| newsgroup. I'll try to extend my heap as you suggested Bjorn - I do see a
| lot of GC in the Eden space - it's wild.

it looks like your Eden space may be a bit small (I don't know what
your application does, so I can't really give you very precise
advice). I did however remember that you had some parameters to
choose the parallell GC and that some other memory tuning options.

the general advice is to start with the "coarse grained" tuning
parameters -- start by just enlarging the heap size. RAM is cheap and
the JVM is a lot happier when it has a bit of room. the next thing to
do may be to tune the ratio of eden space (young generation) to the
rest of the heap etc.

switching GC algorithms is not something I have a lot of experience
doing since I usually am able to get the behavior I want just by doing
really simple tweaks and looking at visual GC.

in any case, having visual GC you can now get a better idea of what
the system is doing, so it should be possible to figure out when you
are making changes that have a positive effect .

-Bjørn

 

[Bjorn Borud]

[Wojtek <[email protected]>]
|
| It is a Web application. When no hits are happening, it should be
| doing nothing. And my code does no networking.
|
| Mind you I do have several timers running, but then the memory use
| goes from 30M to 74M over a 2 hour period, then drops back to 30M,
| which seems kind of excessive for timers.

you're talking about the Eden space, right? 44Mb allocated in a 2 hour
period isn't really dramatic enough to warrant frenetic "optimization"
efforts for an idle app, in my opinion. if Eden is collected only
every 2 hours there is next to no cost associated with those objects.
if you really want to reduce memory footprint, make sure that you keep
eden large enough to cope comfortably during high load.

44Mb is a trivial amount of memory if you compare to memory hogs like
Firefox .

-Bjørn

 

[Tom Hawtin]

Is it really idle? If it's got a GUI all kinds of short-lived FooEvent

Well, no it's not. It presumably has instrumentation attached to it. The
effect is quite normal if you are using something like jconsole.

Tom Hawtin

 

[Wojtek]

Bjorn Borud wrote :

I have a Web app where I capture the used memory every ten
minutes. The result is displayed as a bar graph. Even when the app is
idle, the used memory looks like a saw tooth:

* * *
** ** **
*** *** ***
*************

provided that you have an adequate sampling frequency, which in this
case would be at least two samples or more in the time between minor
collections, a distinct sawtooth shape is usually what you want for
programs like web apps. the period of the sawtooth should be some
multiple of the request time for the great majority of requests.
(which should be simple unless you have obscene variance).

No, this is a totaly idle effect. This is observed on the test server
where no one is using it over-night. Logs prove that no hits were made.

With activity the pattern is much more random, with spikes as graphics
are processed. It takes a LOT of memory to rotate a 3Mbyte image.

 

[Wojtek]

Twisted wrote :

5 or 6 bytes per millisecond on average. One new object created each
millisecond on average is more than enough to cause what you observe.
If the timers have resolutions of a few ms, there are a few of them,
and they generate new Dates each time they update ... there you go.

The memory stat timer triggers every ten mnutes. The others are set for
24 hours.

 

[Bjorn Borud]

[Wojtek <[email protected]>]
|
| No, this is a totaly idle effect. This is observed on the test server
| where no one is using it over-night. Logs prove that no hits were
| made.

well, there is obviously some periodic activity and the sampling of
values itself probably causes objects to be created and dropped.

| With activity the pattern is much more random, with spikes as graphics
| are processed. It takes a LOT of memory to rotate a 3Mbyte image.

if you get a high rate of minor collections you may need more memory
allocated to the heap?

-Bjørn

 

[Wojtek]

Bjorn Borud wrote :

No, this is a totaly idle effect. This is observed on the test server
where no one is using it over-night. Logs prove that no hits were
made.

well, there is obviously some periodic activity and the sampling of
values itself probably causes objects to be created and dropped.

With activity the pattern is much more random, with spikes as graphics
are processed. It takes a LOT of memory to rotate a 3Mbyte image.

if you get a high rate of minor collections you may need more memory
allocated to the heap?

It is set to 2GBytes on the test machine, 1GBytes on my dev machine.

Spikes is relative. If I see use go from 8% to 20%, then back to 2%,
that 20% is a spike.

The highest I ever saw it was 80%, after several image manipulations in
a short time. The next image process caused GC to run, and the usage
dropped to 2%, then up again as more images were processed. On my dev
machine, done by myself during testing.

The usual idle is from 2% to 9%, back to 2%.

I am not worried at all about the actual memory used, just curios about
the use pattern. It may very well be that my app is not causing it at
all, but rather Tomcat doing something periodically, then GC runs.

 

[Twisted]

I am not worried at all about the actual memory used, just curios about
the use pattern. It may very well be that my app is not causing it at
all, but rather Tomcat doing something periodically, then GC runs.

If the app is running on the same JVM as some additional code, it's
very likely that code isn't completely idle when it's "idle". To get
the observed behavior you would not need a very high rate of object
creation, depending on the sizes of the objects. Even small objects at
the glacial pace of one every few million clock cycles is sufficient
to rack up 44MB of garbage in two hours.

As long as the sawtooth is level rather than trending upward I
wouldn't worry. If it's trending upward it's accumulating cruft
somewhere and is doomed to eventually die of OOME, probably because a
permanently-reachable collection is growing without bound. But it
doesn't look like that's happening in your case. (If it ever does,
identify the culprit collection. If it's a Map consider using
WeakHashMap. If it's anything else consider changing it from Set<Foo>
or List<Foo> or whatever to <WeakReference<Foo>> and holding a direct
reference to each Foo somewhere else that is discarded once that Foo
is no longer useful. Each WeakReference must be created on a
ReferenceQueue which must be occasionally polled to remove references
from the queue, and remove any that are found also from the
collection. A thread can sleep(1000) then lock and poll the reference
queue until empty, then lock the collection and remove all the polled
reference objects, then sleep again. In fact, there really should be
Soft and WeakFoo collection wrapper classes that do this for you --
hold an internal ReferenceQueue, wrap added items in a reference
object on that queue, are synchronized, and collectively use a worker
thread pool to periodically clean out their queues and dead reference
objects, while also handling dereferencing objects and releasing
resources. Heck maybe I should write and publish WeakList and WeakSet
wrappers ... I may need some myself some day anyway. Soft versions
would make good caches; Weak versions where you want to store
references to objects live elsewhere in a system for whatever reason
but not to otherwise-dead objects, and the collection isn't a map's
keys. A map with weak values might be useful too, for that matter...

 

[Bjorn Borud]

[Wojtek <[email protected]>]
|
| The highest I ever saw it was 80%, after several image manipulations
| in a short time. The next image process caused GC to run, and the
| usage dropped to 2%, then up again as more images were processed. On
| my dev machine, done by myself during testing.

how often do minor and major GCs, respectively, run during this? does
it run minor GCs several times per second?

-Bjørn