Default timezone changes out of nowhere?

[Werner Lehmann]

Hi everybody,

we have a very serious problem with timezones. Basically the default
timezone is changing for no apparent reason back to UTC, but only after
some time has lapsed since application start. This totally messes up any
date/time output and calculations.

To investigate this we are logging the default tiemzone, timezone offset
and daylight saving offset frequently with Timezone.getDefault(). For
some hours it would be 'America/Denver' with proper offsets, then
suddenly it changes to UTC. Nobody is ever calling
Timezone.setDefault(). There is no apparent reason or pattern why or
when it happens.

This is on a BEA Weblogic 8.1 app server with an RMI server, a few
servlets and JSPs in a 1.4.2 JDK.

Any ideas what could be causing this? Is it somehow possible to tap into
the JDK method Timezone.setDefault(), e.g. to log a stack trace, in
order to learn what is executing this? Your help is greatly appreciated,
after more than a week I can't think of anything else to try...

Werner

 

[Andrew Thompson]

we have a very serious problem with timezones. Basically the default
timezone is changing for no apparent reason back to UTC, but only after
some time has lapsed since application start. This totally messes up any
date/time output and calculations.

What happens if you call TimeZone.setDefault(null)
in that instance? Does the timezone revert to Amrc/Dnvr,
or is it stuck at UTC?

...Is it somehow possible to tap into
the JDK method Timezone.setDefault(), e.g. to log a stack trace, in
order to learn what is executing this?

Explicitly check for the change in a separate thread and
throw a new Exception() when it occurs?

 

[Werner Lehmann]

Hi Andrew,

What happens if you call TimeZone.setDefault(null)
in that instance? Does the timezone revert to Amrc/Dnvr,
or is it stuck at UTC?

I see, calling setDefault() with a null parameter will re-init the
timezone from the OS again - unless this mechanism does not work anymore
or the tz on the OS has become UTC, too.

Explicitly check for the change in a separate thread and
throw a new Exception() when it occurs?

We already see in the log4j log when the change occurs. Throwing an
exception does not achieve anything else, does it? I mean we still do
not know who called .setDefault(). I was hoping maybe there was some
debugging tool available which could do this.

Admittedly the new thread might poll more often and detect the change
sooner. We are not even sure which one of the RMI server, 3rd party
jars, servlets... etc could be causing this, and to further add to the
trouble, it only occurs in the production environment.

Thanks for your suggestions, we will try those.

Werner

 

[Raymond DeCampo]

Hi Andrew,




I see, calling setDefault() with a null parameter will re-init the
timezone from the OS again - unless this mechanism does not work anymore
or the tz on the OS has become UTC, too.



We already see in the log4j log when the change occurs. Throwing an
exception does not achieve anything else, does it? I mean we still do
not know who called .setDefault(). I was hoping maybe there was some
debugging tool available which could do this.

Well, a debugger could put a breakpoint on the method if that's what you
desire.

Admittedly the new thread might poll more often and detect the change
sooner. We are not even sure which one of the RMI server, 3rd party
jars, servlets... etc could be causing this, and to further add to the
trouble, it only occurs in the production environment.

Are you sure that the value returned by the method can only be changed
from within the JVM? What I'm getting at is, is it possible some other
process is changing some OS level setting and that is being propagated
down into the JVM?

Ray

 

[Werner Lehmann]

Ray,

Well, a debugger could put a breakpoint on the method if that's what you
desire.

even if this only happens within some WebLogic JVM on production? Can a
debugger (which one?) attach to the running Java process?

Or for that matter, are there any tools which can inspect the currently
running JVMs, get their system properties, show what they are executing
at that time, maybe even find out about their timezone setting? That
would be a tremendous help.

Are you sure that the value returned by the method can only be changed
from within the JVM? What I'm getting at is, is it possible some other
process is changing some OS level setting and that is being propagated
down into the JVM?

I am not sure at all what changes the timezone. The only way I know what
*could* cause it is the TimeZone.setDefault() method. If there are other
ways, please tell me. We can rule out a change on the OS level after the
Java app has started. A change on the OS level is not propagated to Java
process and in the other direction, the change of the default timezone
is not influencing the OS timezone. Furthermore,
System.currentTimeMillis always returns UTC time even if the default
timezone (in the JVM) is wrong. In other words,

println(currentTimeMillis)
TimeZone.setDefault(UTC)
println(currentTimeMillis)

outputs the same millis value.

Werner

 

[Raymond DeCampo]

Ray,




even if this only happens within some WebLogic JVM on production? Can a
debugger (which one?) attach to the running Java process?

Every Java debugger I have ever worked with has had the ability to
attach remotely to a Java process. It requires that the process be
started with particular flags however.

Eclipse is free and likely has this capability. I'd be wary of hooking
up a debugger to your production system however.

Or for that matter, are there any tools which can inspect the currently
running JVMs, get their system properties, show what they are executing
at that time, maybe even find out about their timezone setting? That
would be a tremendous help.



I am not sure at all what changes the timezone. The only way I know what
*could* cause it is the TimeZone.setDefault() method. If there are other
ways, please tell me. We can rule out a change on the OS level after the
Java app has started. A change on the OS level is not propagated to Java
process

How do you know this? It is conceivable that the implementation of
TimeZone.getDefault() makes a system call to the OS.

and in the other direction, the change of the default timezone
is not influencing the OS timezone. Furthermore,
System.currentTimeMillis always returns UTC time even if the default
timezone (in the JVM) is wrong. In other words,

println(currentTimeMillis)
TimeZone.setDefault(UTC)
println(currentTimeMillis)

outputs the same millis value.

Werner

Ray

 

[Werner Lehmann]

I think we found the culprit, finally. Must still wait for confirmation
that this fixes it though.

It seems to be the Oracle 9 JDBC driver. To read a java.sql.Timestamp
from a ResultSet you can use getTimestamp(int). But there is also an
overloaded method getTimestamp(int, Calendar) which you can use to
specify the timezone which is to be applied. Here we use an UTC calendar
to avoid any automatic conversions.

This works fine, but how Oracle did it is *evil*. The getTimestamp call
first sets the default timezone to UTC (or whatever timezone the
calendar parameter uses), then gets the date from the DB and then resets
the default timezone to what is was before.

While this is awful enough, it is not even threadsafe because the global
default timezone is as bad as a static field, when it comes to thread
safety. If two threads read dates concurrently from the DB it can happen
that the second thread starts with this when the default timezone has
just been changed to UTC by the first thread. Now, if the second thread
finishes last it will revert to UTC again because it thinks this was the
timezone which was active before. The result is that the timezone stays
at UTC from now on. What a mess.

BTW, Oracle seems to have fixed this with the Oracle 10 driver. But this
driver behaves differently with CLOB fields. It is not API compatible.
Plus, customer prefers not to use it with an Ora9 database...

Werner