WebSphere ClassCastException - finds jar by magic

[Jerry]

My group is using WebSphere 4.0.6 on Solaris. We have tried to use a
new version of xerces and xalan in an unrelated (non-WebSphere,
non-J2EE, plain old public static void main app). We placed the newer
versions in a directory containing various other libraries (log4j,
avalon, velocity, etc.)

The various other libraries I have mentioned are listed in various
manifests in jar and war files. For example, one of our war file's
META-INF/MANIFEST.MF looks something like this:

Manifest-Version: 1.0
Created-By: 1.3.1_09 (Sun Microsystems Inc.)
Class-Path: /foo/bar/blort/viking.jar
/foo/bar/blortejb/wpejb.jar
/foo/bar/lib/java/struts.jar
/foo/bar/lib/java/avalon-framework-4.1.4.jar
/foo/bar/lib/java/ecs-1.4.2.jar
/foo/bar/lib/java/fop-0.20.5.jar
/foo/bar/lib/java/commons-collections-3.0.jar
/foo/bar/lib/java/log4j-1.2.8.jar
/foo/bar/lib/java/velocity-1.4.jar
/opt/IBMdb2/V7.1/java12/db2java.zip

Nowhere do we reference the directory /foo/bar/lib as such; it is
always a specific jar file.

Nevertheless, as soon as we dropped new versions of xalan and xerces
into /foo/bar/lib, we began getting a ClassCastException out of
WebSphere. Three developers have examined the WebSphere documentation
on class loader order, and every conceivable place where the directory
could be specified in a classpath, and we can find no reason for it to
recognize these files at all. But when we take them away, the problem
disappears.

Does anyone have any idea why it is "discovering" these?

Thanks in advance.

Jerry Oberle
perl -e 'printf "mailto%c%s%c%s%cjpmchase%ccom%c", 58, "Gerard", 46,
"Oberle", 64, 46, 10;'

 

[Ben_]

If I understand correctly, it's a non-J2EE application that's running in a
JVM of its own and adding a jar to a given directory makes it appear on the
classpath ?

As it's a non-J2EE application, it means you have a script of your own to
launch the application.

Did you check that it does not use a convenience feature to enumerate
directories to place all jar's it find on the classpath ?

 

[Jerry]

Not quite.

We have a non-J2EE application that needs the new xerces & xalan. That
application explicitly adds the two new jar files to its classpath, and
runs just fine.

However, our WebSphere application server, that should know nothing
about these two jar files, appears to load classes from them anyway,
and we do not want it to. The problem is that we cannot figure out how
WebSphere's class loaders are "discovering" these new xalan and xerces
classes.

Thanks.

Jerry Oberle
perl -e 'printf "mailto%c%s%c%s%cjpmchase%ccom%c", 58, "Gerard", 46,
"Oberle", 64, 46, 10;'

 

[Ben_]

OK.

And in which directory did you place the jar files that WebSphere sees ?

Also, are there system properties defined on the Application Server (like
ws.ext.dirs) ?

 

[Jerry]

Thank you for your help.

The jar files it appears to see are in /foo/bar/lib (not its real name,
of course), but the jar files are not explicitly listed in any of the
manifests that we've written.

No, we do not have ws.ext.dirs defined. We have many system properties
defined, but they are all things used by our application code, because
certain guilty parties who shall remain nameless profess J2EE
expertise, but have never heard of java:comp/env But I digress...

We have another interesting discovery. When we renamed
/foo/bar/lib/xercesImpl.jar to /foo/bar/lib/blort1.jar and
/foo/bar/lib/xml-apis.jar to /foo/bar/lib/blort2.jar, WebSphere seemed
to ignore them. That is, the ClassCastException went away.

At this point, someone suggested simply running our separate
application with /foo/bar/lib/blort1.java and /foo/bar/lib/blort2.jar
in its jar file's manifest, and forget the problem. Three to one, we
voted to continue investigating until we understood what was happening.

I think this latest test suggests that, indeed, there exists somewhere
a specific reference to either xml-apis.jar and/or xercesImpl.jar which
we have failed to find. It does not seem likely that such a reference
exists in any of the jars shipped with WebSphere 4.0.6, because these
libraries weren't packed under those names in those days. It further
seems unlikely that anything we received from outside the company
included a fully qualified path name that began with /foo/bar/lib.

I can well imagine someone posting a response that says the inescapable
conclusion to be drawn is that someone in our own shop must have coded
a specific reference to these somewhere, and we're simply idiots
because we can't find it. But I'm hanging on to hope that there is
another answer, because I'm very disconcerted by any sort of scientific
proof that I'm an idiot.

Thanks in advance for any brilliant --- or even stupid --- ideas or
suggestions.

Jerry Oberle
perl -e 'printf "mailto%c%s%c%s%cjpmchase%ccom%c", 58, "Gerard", 46,
"Oberle", 64, 46, 10;'

 

[Chris Uppal]

The jar files it appears to see are in /foo/bar/lib (not its real name,
of course), but the jar files are not explicitly listed in any of the
manifests that we've written. [...]
We have another interesting discovery. When we renamed
/foo/bar/lib/xercesImpl.jar to /foo/bar/lib/blort1.jar and
/foo/bar/lib/xml-apis.jar to /foo/bar/lib/blort2.jar, WebSphere seemed
to ignore them. That is, the ClassCastException went away. [...]
Thanks in advance for any brilliant --- or even stupid --- ideas or
suggestions.

Ok, here's a stupid one: is it possible that someone has symlinked to those jar
files from somewhere where WebSphere /should/ be looking ?

-- chris

 

[Ben_]

Going for another stupid one: I hope "foo/bar" does not stand for
"WebSphere/AppServer"...

 

[Jerry]

Thanks, Chris and Ben.

No, /foo/bar is not WebSphere/AppServer. It's deliberately outside of
the WebSphere install tree, specifically to avoid this sort of problem.
Our theory is that we can explicitly point to the things a given
application requires by putting a specific reference in its
MANIFEST.MF. And that was not at all a stupid question.

As for whether someone has a sym link pointing to those files, I'll go
look. I didn't think of that. If that's the case, you'll be seeing me
handcuffed on the evening news, with pictures of programming or system
administration cadavers being wheeled out behind me on gurneys draped
with sheets.

Jerry Oberle
perl -e 'printf "mailto%c%s%c%s%cjpmchase%ccom%c", 58, "Gerard", 46,
"Oberle", 64, 46, 10;'

 

[Ben_]

Definitely, you need a break... Hopefully, it's almost week-end time ...

 

[Jerry]

Thank you, both Chris and Ben.

And the answer is...

We use an Apache class library somewhere in our application to generate
PDF. The class library is known as FOP; see
http://xmlgraphics.apache.org/fop/0.20.5/index.html Although this is
described as "the first choice for those who want a stable product for
use in production environments," the distributed jar file contains the
following gems in its manifest:

Class-Path: xercesImpl-2.2.1.jar xml-apis.jar xalan-2.4.1.jar batik.jar
jimi-1.0.jar avalon-framework-cvs-20020806.jar

The Avalon code from CVS as of an arbitrary date was especially
reassuring "for use in production environments."

Previously, we either had none of these anywhere, or, in the case of
batik, xerces and xalan had other versions in our classpath anyway, so
the class loaders pulled the various classes from whatever we had,
ignoring the missing files from the fop manifest.

Once the newer xerces and xalan jar files suddenly appeared, they
suddenly started getting loaded and objects passed about to other
objects that were apparently loaded by different classloaders. So the
JVM saw the various flavors as different classes, and refused to cast
an instance of one to an instance of the other.

The solution was to pull the source for fop, change the build.xml to
remove the unnecessary trash from its manifest, and build our own. No
changes to the source were necesssary.

Jerry Oberle
perl -e 'printf "mailto%c%s%c%s%cjpmchase%ccom%c", 58, "Gerard", 46,
"Oberle", 64, 46, 10;'

 

[Ben_]

That was a nasty one...

 

[Chris Uppal]

And the answer is...

Thanks for the update. I was wondering if you were under police detention
somewhere ;-)

-- chris