[OT?] Using classfileset in Ant

[Rhino]

I suppose this is largely offtopic since it is not specifically a Java
question but it relates to Java code and I haven't been able to get an
answer on the appropriate Ant mailing list so I hope you can forgive me for
asking here.

---

I'm trying to use classfileset to build a jar for the first time but it
doesn't seem to be recursing the way I expected.

This is my target:
<!--==============================================================

Jar everything together into one big jar.

==============================================================-->

<target name="Jar_all" description="Jar all classes that are needed by Foo
project.">

<classfileset id="reqdClasses" dir="${bin.dir}">

<root classname="mydomain.foo.Foo"/>

</classfileset>

<delete file="${jar.dir}/All.jar" description="delete old jar"/>

<jar destfile="${jar.dir}/All.jar" description="create new jar">

<fileset refid="reqdClasses"/>

</jar>

</target>


When I run it, the jar contains only three class files (and a manifest):

mydomain.foo.Foo.class
mydomain.foo.FooConstants.class
mydomain.foo.FooPreferences.class

mydomain.foo.Foo is indeed the main class of the program and it is indeed
dependent on mydomain.foo.FooConstants and mydomain.foo.FooPreferences.

Unfortunately, none of the classes that *they* depend on are present in the
jar. There are quite a few of them, all belonging to packages starting with
mydomain.common.

I assume that I've written the target incorrectly and have omitted something
that tells Ant to recurse to find the dependencies of the dependencies but I
can't find any clarification of this in the manual.

Could someone please confirm that the classfileset *does* find dependencies
recursively and explain what I need to change in my target to make the
recursion take place?

If classfileset does not recurse, can anyone suggest any other way in which
Ant could be used to build a jar containing my main class and *ALL* of its
dependents, not just the immediate 'children' but all of the
'grandchildren', 'great-grandchildren' etc.?



--
Rhino
---
rhino1 AT sympatico DOT ca
"There are two ways of constructing a software design. One way is to make it
so simple that there are obviously no deficiencies. And the other way is to
make it so complicated that there are no obvious deficiencies." - C.A.R.
Hoare

 

[John C. Bollinger]

I'm trying to use classfileset to build a jar for the first time but it
doesn't seem to be recursing the way I expected.

This is my target:
<!--==============================================================

Jar everything together into one big jar.

==============================================================-->

<target name="Jar_all" description="Jar all classes that are needed by Foo
project.">

<classfileset id="reqdClasses" dir="${bin.dir}">

<root classname="mydomain.foo.Foo"/>

</classfileset>

<delete file="${jar.dir}/All.jar" description="delete old jar"/>

<jar destfile="${jar.dir}/All.jar" description="create new jar">

<fileset refid="reqdClasses"/>

</jar>

</target>

Looks fine to me.

When I run it, the jar contains only three class files (and a manifest):

mydomain.foo.Foo.class
mydomain.foo.FooConstants.class
mydomain.foo.FooPreferences.class

mydomain.foo.Foo is indeed the main class of the program and it is indeed
dependent on mydomain.foo.FooConstants and mydomain.foo.FooPreferences.

Unfortunately, none of the classes that *they* depend on are present in the
jar. There are quite a few of them, all belonging to packages starting with
mydomain.common.

You need to have all the class files available in the fileset's base
directory in order for the ClassFileSet to find and analyze them.
Remember that although Ant permits you to use multiple source
directories and a complex classpath for Java compilation, its FileSet
type (and derivatives) specifically refers to a collection of files all
relative to some common base directory. My first guess is that this is
where you are falling down.

I assume that I've written the target incorrectly and have omitted something
that tells Ant to recurse to find the dependencies of the dependencies but I
can't find any clarification of this in the manual.

Could someone please confirm that the classfileset *does* find dependencies
recursively and explain what I need to change in my target to make the
recursion take place?

Here is an example from one of my projects, used successfully for quite
some time with Ant 1.5.x:

<target name="recipnetd.jar-files" depends="compiled-java_src-site">
<classfileset id="recipnetd.jar-files" dir="${build-class-dir}">
<rootfileset dir="${build-class-dir}">
<patternset refid="recipnetd-root_class-patterns" />
</rootfileset>
</classfileset>
</target>

This target just creates the ClassFileSet for later use by a different
target, but it does it quite successfully. The classes included come
from a complex package hierarchy, and some of them are in branches of
the directory tree that are parallel to those in which the root classes
reside. I would not expect the difference between using a <root> and
using a <rootfileset> to matter, but I cannot knowledgeably claim that
it in fact doesn't.


John Bollinger
(e-mail address removed)

 

[Rhino]

Looks fine to me.


You need to have all the class files available in the fileset's base
directory in order for the ClassFileSet to find and analyze them.
Remember that although Ant permits you to use multiple source
directories and a complex classpath for Java compilation, its FileSet
type (and derivatives) specifically refers to a collection of files all
relative to some common base directory. My first guess is that this is
where you are falling down.

I think you're right; something about my original formulation of the
classfileset seems to limit the scope of the search for dependencies to the
same directory that contains the 'root classname'. Unfortunately, I haven't
figured out how to specify *two* directories in the classfileset definition;
I tried doing a 'dirset' but got a message that dirset wasn't supported in
classfileset.

But maybe I just did it wrong; I find the manual articles on some of these
topics, like dirset, more mystifying than enlightening. The examples, when
present are weak, incomplete, and poorly explained.

Here is an example from one of my projects, used successfully for quite
some time with Ant 1.5.x:

<target name="recipnetd.jar-files" depends="compiled-java_src-site">
<classfileset id="recipnetd.jar-files" dir="${build-class-dir}">
<rootfileset dir="${build-class-dir}">
<patternset refid="recipnetd-root_class-patterns" />
</rootfileset>
</classfileset>
</target>

This target just creates the ClassFileSet for later use by a different
target, but it does it quite successfully. The classes included come
from a complex package hierarchy, and some of them are in branches of
the directory tree that are parallel to those in which the root classes
reside. I would not expect the difference between using a <root> and
using a <rootfileset> to matter, but I cannot knowledgeably claim that
it in fact doesn't.

You have no idea how much it helps to know that classfileset is supposed to
recurse! I posted two or three different posts on the ant-user mailing list
asking about classfileset and no one answered. I really wasn't sure if
classfileset was even supposed to recurse, although it seemed logical that
it should.

Your example has helped me move forward - I think. By imitating it, I have
greatly increased the number of files that my jar picked up which *feels*
like a step forward. You didn't include your definition of the
'recipnetd-root_class-patterns' patternset so I took my best shot and wrote
this:

<patternset id="ps.all" description="All of the source/class directories
used by Foo project.">

<include name="Common\**"/>

<include name="Foo\**"/>

</patternset>

<classfileset id="reqdClasses2" dir="${workspace.dir}">

<rootfileset dir="${workspace.dir}"/>

<patternset refid="ps.all"/>

</classfileset>

<delete file="${jar.dir}/All2.jar" description="delete old jar"/>

<jar destfile="${jar.dir}/All2.jar" description="create new jar">

<fileset refid="reqdClasses2"/>

</jar>

Unfortunately, the jar now contains all of the files in the Common and Foo
projects, even classes that have nothing to do with Foo, i.e. classes that
Foo.class has no dependencies on. I clearly don't understand this whole
classfileset/patternset thing yet: I was hoping that the patternset was
being used to set the scope of directories thru which classfileset could
search for dependencies of my main application class, Foo.class.
Unfortunately, it doesn't seem to have worked that way at all.

Of course, I no longer have anything in the Ant code that says I should
start with mydomain.foo.Foo.class so that is likely the source of the
problem. However, I don't see how/where I could specify that now; there
isn't a logical place to put the 'root classname' line now. Or is there?

Do you have any idea how I can make this work? If not, could you possibly
post your definition of your patternset and maybe I can figure it out from
there.

Rhino

 

[John C. Bollinger]

I think you're right; something about my original formulation of the
classfileset seems to limit the scope of the search for dependencies to the
same directory that contains the 'root classname'. Unfortunately, I haven't
figured out how to specify *two* directories in the classfileset definition;
I tried doing a 'dirset' but got a message that dirset wasn't supported in
classfileset.

You cannot specify two directories. Read again what I wrote: "[Ant's]
FileSet type (and derivatives) specifically refers to a collection of
files *all relative to some common base directory*." (Emphasis added.)
In this case that should be the (base) compilation target directory.

You have no idea how much it helps to know that classfileset is supposed to
recurse! I posted two or three different posts on the ant-user mailing list
asking about classfileset and no one answered. I really wasn't sure if
classfileset was even supposed to recurse, although it seemed logical that
it should.

I'm glad it helped.

Your example has helped me move forward - I think. By imitating it, I have
greatly increased the number of files that my jar picked up which *feels*
like a step forward. You didn't include your definition of the
'recipnetd-root_class-patterns' patternset so I took my best shot and wrote

No, I did not include that definition, because it is specific to my
application and not relevant to this discussion. The example indicates
that it refers to a patternset, and its context shows that it is used to
specify the root classes for the ClassFileSet. Note, by the way, that
these are supposed to be the *root* classes -- the starting points for
the dependency analysis.

this:

<patternset id="ps.all" description="All of the source/class directories
used by Foo project.">

<include name="Common\**"/>

<include name="Foo\**"/>

</patternset>

So your pattern set contains every file under Common or Foo. That is
unlikely to be what you want.

<classfileset id="reqdClasses2" dir="${workspace.dir}">

<rootfileset dir="${workspace.dir}"/>

<patternset refid="ps.all"/>

</classfileset>

<delete file="${jar.dir}/All2.jar" description="delete old jar"/>

<jar destfile="${jar.dir}/All2.jar" description="create new jar">

<fileset refid="reqdClasses2"/>

</jar>

Unfortunately, the jar now contains all of the files in the Common and Foo
projects, even classes that have nothing to do with Foo, i.e. classes that
Foo.class has no dependencies on.

Of course it does. You specified them all as root classes.

I clearly don't understand this whole
classfileset/patternset thing yet: I was hoping that the patternset was
being used to set the scope of directories thru which classfileset could
search for dependencies of my main application class, Foo.class.
Unfortunately, it doesn't seem to have worked that way at all.

It's impossible to say whether it did or not. Since you included as
root classes everything that the ClassFileSet might otherwise have had
to search for, there's no way to tell whether or not it actually
performed the search.

Of course, I no longer have anything in the Ant code that says I should
start with mydomain.foo.Foo.class so that is likely the source of the
problem. However, I don't see how/where I could specify that now; there
isn't a logical place to put the 'root classname' line now. Or is there?

You really don't appear to need a patternset to define the root classes,
since there is only one, but as long as you're doing that how about
trying a patternset that matches only mydomain/foo/Foo.class?


John Bollinger
(e-mail address removed)

 

[Rhino]

I think you're right; something about my original formulation of the
classfileset seems to limit the scope of the search for dependencies to the
same directory that contains the 'root classname'. Unfortunately, I haven't
figured out how to specify *two* directories in the classfileset definition;
I tried doing a 'dirset' but got a message that dirset wasn't supported in
classfileset.

You cannot specify two directories. Read again what I wrote: "[Ant's]
FileSet type (and derivatives) specifically refers to a collection of
files *all relative to some common base directory*." (Emphasis added.)
In this case that should be the (base) compilation target directory.

To my way of thinking, my original formulation of the target made perfect
sense:

<classfileset id="reqdClasses" dir=${bin.dir}">
<root classname="mydomain.foo.Foo"/>
</classfileset>

I was trying to say "please build a jar containing mydomain.foo.Foo.class
and all of its direct and indirect dependencies". I assumed that the 'dir'
parameter of 'classfileset' identified the folder in which the package
'mydomain.foo' could be found.

However, when I ran that task, I found that the jar contained only
mydomain.foo.Foo's immediate dependents, mydomain.foo.FooConstants and
mydomain.foo.FooPreferences, and none of *their* dependents. Two
possibilities crossed my mind:
a) Maybe classfileset was not designed to recurse and was 'working as
designed' by returning only the immediate dependencies of the class that I
had specified as the root.
b) Something in my classfileset definition told classfileset to limit the
scope of its recursion. The only feasible candidate seemed to be 'dir' so it
struck me that *maybe* 'dir' somehow set the scope of the recursion and if I
could somehow add the name of the directory containing the rest of the code
to the 'dir' parameter, classfileset would be able to find the other
dependencies.

I'm glad it helped.
wrote

No, I did not include that definition, because it is specific to my
application and not relevant to this discussion. The example indicates
that it refers to a patternset, and its context shows that it is used to
specify the root classes for the ClassFileSet. Note, by the way, that
these are supposed to be the *root* classes -- the starting points for
the dependency analysis.


So your pattern set contains every file under Common or Foo. That is
unlikely to be what you want.


Of course it does. You specified them all as root classes.

Fair enough. As I've said, I don't find the documentation on classfileset
very clear so I was trying everything that seemed reasonable to make
classfileset recurse beyond the immediate dependents of my root class.
Apparently, including that patternset - in which I was only imitating your
own example - did not increase the recursion at all, it simply increased the
number of root classes.

So, I'm back to Square One then and both of my original theories are still
possible:
- maybe classfileset can only find direct dependents and can't recurse. This
is seeming increasingly likely since I can't see any flag that is telling it
not to recurse; therefore, maybe it IS "working as designed". If this is the
case, then it's either a fairly useless tool because it really OUGHT to
recurse, at least in my view, or I've got a fundamental design flaw in the
way I've designed my classes: maybe having dependents with their own
dependents is fundamentally wrong.
- maybe I AM telling classfileset not to recurse without realizing it. If
this is the case, I'd love to know how to tell it that it should recurse
into other packages and directories.

Can you - or anyone else following this thread - please confirm or deny
categorically that classfileset WILL recurse into other packages and/or
directories to find all direct and indirect dependencies of the root class?
And, if this recursion into other packages and directories IS possible, how
do I get classfileset to do it, because it certainly isn't doing it now
despite my best efforts.

Rhino

 

[John C. Bollinger]

You cannot specify two directories. Read again what I wrote: "[Ant's]
FileSet type (and derivatives) specifically refers to a collection of
files *all relative to some common base directory*." (Emphasis added.)
In this case that should be the (base) compilation target directory.

To my way of thinking, my original formulation of the target made perfect
sense:

<classfileset id="reqdClasses" dir=${bin.dir}">
<root classname="mydomain.foo.Foo"/>
</classfileset>

And I responded that that looked good.

I was trying to say "please build a jar containing mydomain.foo.Foo.class
and all of its direct and indirect dependencies". I assumed that the 'dir'
parameter of 'classfileset' identified the folder in which the package
'mydomain.foo' could be found.

Perhaps now we're getting somewhere. The dir attribute should identify
the base directory of the _fileset_. *All* the classes that you want in
the fileset must be somewhere in the subtree rooted at that directory.
(How else would Ant know where to find them?) I wrote earlier that it
looked like perhaps not all of the class files were available, and it's
beginning to sound as if I was right. Note also that for a ClassFileSet
in particular, the class files should be arranged relative to the
fileset's base directory in a tree corresponding to their package names.

So, I'm back to Square One then and both of my original theories are still
possible:
- maybe classfileset can only find direct dependents and can't recurse. This
is seeming increasingly likely since I can't see any flag that is telling it
not to recurse; therefore, maybe it IS "working as designed". If this is the
case, then it's either a fairly useless tool because it really OUGHT to
recurse, at least in my view, or I've got a fundamental design flaw in the
way I've designed my classes: maybe having dependents with their own
dependents is fundamentally wrong.

ClassFileSet follows dependencies recursively. I know this both from
practical experience and from looking at the code. There is nothing
wrong with multiple levels of dependency; indeed, it would be difficult
to write an application of any size without such.

- maybe I AM telling classfileset not to recurse without realizing it. If
this is the case, I'd love to know how to tell it that it should recurse
into other packages and directories.

No, there is no way to tell classfileset to not recurse. But you may
have limited the classes that you will permit it to traverse by your
selection of its base directory, or by the class files actually present
when the dependency analysis occurs. It is extremely likely that your
problem is along these lines.


John Bollinger
(e-mail address removed)

 

[Rhino]

You cannot specify two directories. Read again what I wrote: "[Ant's]
FileSet type (and derivatives) specifically refers to a collection of
files *all relative to some common base directory*." (Emphasis added.)
In this case that should be the (base) compilation target directory.

To my way of thinking, my original formulation of the target made perfect
sense:

<classfileset id="reqdClasses" dir=${bin.dir}">
<root classname="mydomain.foo.Foo"/>
</classfileset>

And I responded that that looked good.

I was trying to say "please build a jar containing mydomain.foo.Foo.class
and all of its direct and indirect dependencies". I assumed that the 'dir'
parameter of 'classfileset' identified the folder in which the package
'mydomain.foo' could be found.

Perhaps now we're getting somewhere. The dir attribute should identify
the base directory of the _fileset_. *All* the classes that you want in
the fileset must be somewhere in the subtree rooted at that directory.
(How else would Ant know where to find them?) I wrote earlier that it
looked like perhaps not all of the class files were available, and it's
beginning to sound as if I was right. Note also that for a ClassFileSet
in particular, the class files should be arranged relative to the
fileset's base directory in a tree corresponding to their package names.

This is clearly the main point of confusion. Am I correct in understanding
that the 'dir' attribute of the classfileset needs to point to the directory
where the root class can be found? That would make sense to me; as you said,
how else can Ant hope to find it. Now, the million dollar question: does the
'dir' attribute limit the scope of the recursion of the dependencies in any
way???

I have correctly specified the location of the root class in my 'dir'
attribute and I have correctly specified the name of the root class.
However, most of the dependencies of my root class reside either in
different packages beneath the directory named in the 'dir' attribute or
beneath a different directory altogether. Is the Classfileset capable of
recursing to other packages within the specified dir and into other packages
beneath another directory? I *want* classfilesets to be able to recurse into
other packages within the same directory and into other directories but
maybe I'm asking too much.

ClassFileSet follows dependencies recursively. I know this both from
practical experience and from looking at the code. There is nothing
wrong with multiple levels of dependency; indeed, it would be difficult
to write an application of any size without such.

Okay, I'm glad to hear that classfileset really can recurse. I had hoped so
but was beginning to wonder....

The question is: can it recurse into directories that are not the one named
in the 'dir' attribute?

No, there is no way to tell classfileset to not recurse. But you may
have limited the classes that you will permit it to traverse by your
selection of its base directory, or by the class files actually present
when the dependency analysis occurs. It is extremely likely that your
problem is along these lines.

Let me get very specific about my situation, just so that we are both very
clear on what is happening.

The main new code that I have written is in directory 'foo' which contains
two packages, 'mydomain.foo' and 'mydomain.foo.Resources'. Package
mydomain.foo contains three classes:
- Foo
- FooConstants
- FooPreferences
Package mydomain.foo.Resources contains a variety of List, Text, and Message
resource bundles used by class Foo and class FooPreferences.

Class Foo is dependent on all of the items in these two packages. However,
only the List resource bundles actually involve class files; the other two
resource bundle types only result in .properties files.

IN ADDITION, class FooPreferences has direct dependencies on these classes:
- AnimationPanel
- AudioPanel
- DigitalCountdownPanel
- DigitalClockPanel
all of which are in package mydomain.common AND
- LocalizationUtils
- LogUtils
- PreferenceUtils
- ReflectionUtils, all of which are in package mydomain.common.utilities.
All of the packages whose names begin 'mydomain.common' are in a directory
called 'common'.

IN TURN, these classes have dependencies of their own, all of which are also
found in packages beginning with 'mydomain.common'. Therefore, they are all
found in directory 'common' too.

There are a lot of classes in the 'common' directory, many of which are not
needed by class Foo to do its job, and they are organized into a variety of
packages, many of which Foo doesn't need, so I was hoping to construct a jar
that contained only the classes needed by Foo or its dependents. However, I
would be willing to simplify my requirement to having only the packages
needed by Foo to do its job, even if that means having several classes in
the jar that Foo will never use.

Therefore, if classfileset is capable of recursing to other packages in the
same and
different directories, how do I make that happen? Right now, my classfileset
definition -

<classfileset id="reqdClasses" dir="foo">
<root classname="mydomain.foo.Foo"/>
</classfileset>

is returning only these three classes: Foo, FooConstants, and
FooPreferences, in other words the immediate dependents of the root class
that are in the same package. The other immediate dependents that are in
another package in the same directory are NOT being picked up, nor are the
dependents in other packages under the common directory being picked up.

If it is possible to incorporate all direct and indirect dependencies of my
root class in my classfileset, even if those depedent classes reside in
different packages and jars, what do I need to change in my definition to
make that happen? I have tried everything I could think of without success.

Rhino

 

[John C. Bollinger]

This is clearly the main point of confusion.
(1)

Am I correct in understanding
that the 'dir' attribute of the classfileset needs to point to the directory
where the root class can be found? That would make sense to me; as you said,
how else can Ant hope to find it.
(2)

Now, the million dollar question: does the
'dir' attribute limit the scope of the recursion of the dependencies in any
way???

Are you in fact reading what I write? These questions are both
answered, one directly and one indirectly, by the comments you quoted.
In short:

1) No. In general, the dir attribute would need to refer to an ancestor
of the root class' directory. Exactly which one depends on package names.

2) Yes. Only directories in the subtree rooted at the specified
directory are examined. This is an inescapable consequence of the
nature of Ant FileSets. It is generally not a problem if you compile
all your classes into the same base destination directory.


John Bollinger
(e-mail address removed)

 

[Rhino]

Are you in fact reading what I write?

Yes, of course I am. I am echoing things back to make sure that I understand
what you are saying. I don't think I'm there yet but it's not for lack of
trying.

These questions are both
answered, one directly and one indirectly, by the comments you quoted.
In short:

1) No. In general, the dir attribute would need to refer to an ancestor
of the root class' directory. Exactly which one depends on package names.

2) Yes. Only directories in the subtree rooted at the specified
directory are examined. This is an inescapable consequence of the
nature of Ant FileSets. It is generally not a problem if you compile
all your classes into the same base destination directory.

I still don't see what I need to do differently to get my classfileset to
work. I've explained exactly what my situation is and what the directories
are. We've spent literally days going back and forth as I've tried to
clarify just exactly what I need to do. Call me an idiot - I'm not, but if
it helps you, go ahead and think it - but I really am trying to understand
how to make this work.

From what you are saying in response to point 2, I thought I might benefit
by changing my 'dir' attribute to point to 'workspace' the common parent of
both the 'foo' and 'common' directories which, between them, contain all of
the code needed to support my root class. But when I do that, I get
manifest-only jar. I don't know what else to try. I've tried umpteen
variations of this classfileset and I'm STILL not getting anything but the
immediate dependents of the root class - if I get anything at all.

What on earth am I doing wrong?

Rhino

 

[John C. Bollinger]

From what you are saying in response to point 2, I thought I might benefit
by changing my 'dir' attribute to point to 'workspace' the common parent of
both the 'foo' and 'common' directories which, between them, contain all of
the code needed to support my root class. But when I do that, I get
manifest-only jar. I don't know what else to try. I've tried umpteen
variations of this classfileset and I'm STILL not getting anything but the
immediate dependents of the root class - if I get anything at all.

What on earth am I doing wrong?

This is my last shot. All of this is going to be repeated and/or
derived from one and another earlier posts, but perhaps if I repackage
it the situation will become clear. If not, then I recommend that you
study the Ant manual, as part of the problem seems to be that you don't
have a firm grasp of some of the relevant details of Ant data types (not
just ClassFileSet). I cannot do better, or more, than I have now done.

() All the class files that need to be analyzed need to be available at
the time the ClassFileSet is constructed.

() All the class files need to be arranged in an hierarchical directory
tree mirroring the package structure, with a common root directory.

() For ClassFileSet specifically, the class files must be arranged
relative to the common root such that there are no intervening
directories between the root and the lowest-level directories
corresponding to package name segments. This is what you would get in
the compilation target directory, supposing that you compile all the
relevant classes to the _same_ target directory.

() The base directory of the ClassFileSet should be the common root
described in the previous point.


Example:
You need a directory tree similar to this:

classes
|
+-- mycompany
|
+-- common
| |
| +-- subpackage1
|
+-- foo

Directory classes/mycompany/common/subpackage1 contains all the relevant
classes of package mycompany.common.subpackage1 (and may contain any
other classes or files as well). Similarly for each directory in the
subtree.

Once all the classes have been compiled into that directory tree, you
create a ClassFileSet with classes/ as its base directory. Specify as
root classes whichever classes you want to use as starting points for
the dependency analysis. The analysis will be restricted to the
directory subtree rooted at classes/, and (implied, but not previously
stated explicitly) class files will be located in the tree by mapping
their package names to the directory structure.


Good Luck,

John Bollinger
(e-mail address removed)

 

[Rhino]

This is my last shot. All of this is going to be repeated and/or
derived from one and another earlier posts, but perhaps if I repackage
it the situation will become clear. If not, then I recommend that you
study the Ant manual, as part of the problem seems to be that you don't
have a firm grasp of some of the relevant details of Ant data types (not
just ClassFileSet). I cannot do better, or more, than I have now done.

Okay, you're obviously getting tired of this discussion; you think you've
told me everything I need to know and I'm just not getting it. I'm getting
tired of it too because I *am* trying but you never seem to answer my
question about what precisely I have to do differently even though I have
laid out exactly how my code looks and exactly how I've organized the code
(except that I've disguised my real project name with the name 'Foo').

In any case, thank you for your efforts. I really am trying to get to the
bottom of this problem; believe me. I've invested several hours on writing
posts so far (both here and on other newsgroups/mailing lists) and I
wouldn't have done that unless this was really bugging me. On the chance
that you are still following up on this thread and not just washing your
hands of me, here is more information on what I've discovered as a result of
this post.

() All the class files that need to be analyzed need to be available at
the time the ClassFileSet is constructed.

That's not an issue; the compiles were all done before I tried to run the
Ant build. I'm not generating any of the classes within the build.

() All the class files need to be arranged in an hierarchical directory
tree mirroring the package structure, with a common root directory.

They have been all along. I think my problem has been - partly - that I
specified the 'dir' one level too low. More on that in a minute.

() For ClassFileSet specifically, the class files must be arranged
relative to the common root such that there are no intervening
directories between the root and the lowest-level directories
corresponding to package name segments. This is what you would get in
the compilation target directory, supposing that you compile all the
relevant classes to the _same_ target directory.

I now think this is the nub of the problem, at least if I understand it
correctly.

() The base directory of the ClassFileSet should be the common root
described in the previous point.

As stated above, I think I started one level too low in the tree for the
first umpteen attempts. However, when I went up one level to the common
level, I didn't get anything at all.

Example:
You need a directory tree similar to this:

classes
|
+-- mycompany
|
+-- common
| |
| +-- subpackage1
|
+-- foo

Directory classes/mycompany/common/subpackage1 contains all the relevant
classes of package mycompany.common.subpackage1 (and may contain any
other classes or files as well). Similarly for each directory in the
subtree.

Once all the classes have been compiled into that directory tree, you
create a ClassFileSet with classes/ as its base directory. Specify as
root classes whichever classes you want to use as starting points for
the dependency analysis. The analysis will be restricted to the
directory subtree rooted at classes/, and (implied, but not previously
stated explicitly) class files will be located in the tree by mapping
their package names to the directory structure.

I have made some progress with my recursion as a result of the information
in this post. I'm not completely happy yet - it still doesn't seem to be
recursing as far as it should yet and I don't like the implications for how
my code has to be organized - but I am getting more recursion than I was and
I am picking up some of the 'common' classes.

Okay, here's how I did it. I'm going to present my data organization a bit
differently this time because the package names may have inadvertently
confused things in previous posts.

Originally, I had this:

http://www3.sympatico.ca/rhino1/misc/foo_old.jpg and
http://www3.sympatico.ca/rhino1/misc/common.jpg. Both Common and
ExerciseTimer ('foo') were directly beneath the 'workspace' directory.

However, with the original formulation of the classfileset (which you never
criticized) was:

<classfileset id="reqdClasses" dir="${bin.dir}">
<root classname="ca.tonge.exerciseTimer.ExerciseTimer"/>

</classfileset>

With the 'dir' attribute set to workspace\ExerciseTimer, the classfileset
only contained the three classes ExerciseTimer.class,
ExerciseTimerConstants.class, and ExerciseTimerPreferences.class and
wouldn't pick up anything in the Resources directory immediately below
workspace\ExerciseTimer\bin\ca\tonge\exerciseTimer, let alone any of the
dependent classes in the Common branch of the tree.

After reading your post, I tried an experiment and copied all of the common
code beneath ExerciseTimer so that I ended up with:

http://www3.sympatico.ca/rhino1/misc/foo_new.jpg

Therefore, the 'common' and 'exerciseTimer' code is now under the same
directory 'ExerciseTimer'. Now, the classfileset includes much more of the
code, especially the classes under ExerciseTimer\bin\ca\tonge\common.

It's not perfect yet though. On the positive side, all of the classes in the
classfileset appear to be genuine dependents of ExerciseTimer.class, my root
class; none of them appear to have been picked up just because they happened
to be in the same package/directory as a real dependent. On the negative
side, I am still not getting the following: any of the dependent classes in
ExerciseTimer\Resources (ExerciseTimer.class is dependent on everything in
ExerciseTimer\Resources); any of the indirect dependents of the common
classes (I'm getting the dependents which appear in the imports within
ExerciseTimer.class but not any of *their* dependents.)

I'm also far from happy that I had to copy the common code under the
ExerciseTimer directory to get this far; I do NOT want to have to do that
routinely if I can possibly avoid it. I have a separate directory under
workspace for each of my projects; I don't want to have to have a single big
project that contains all of the code I have ever written just to enable
classfileset to find the dependencies. But that is the way I interpreted
what you said and I obviously got a better result than I did earlier.

If you're still following this thread, have I misunderstood you yet again?

I would love to hear you tell me that I can leave the common code under the
Common directory and not have to copy it under the ExerciseTimer directory
and still get full recursion into the common code (as well as the
exerciseTimer/Resources code) - along with instructions on what I have to do
differently in my code! - but I am so tired of this struggle that I am ready
to give up on classfilesets altogether and go back to finding dependencies
manually. The last time only took me two hours, as opposed to the much
longer time I have spent trying to make classfileset work, including the
time it has taken to write several long posts in several different
newsgroups and try every variation of classfileset that I could conceive.

I can't believe how much work it has been to try to get three lousy lines of
code to work. The problem HAS to be something very very simple, probably
related to a conceptual misunderstanding I am having, but I'm just not
seeing it so far.

The only reason I am still banging my head against the wall is the hope that
eventually this will make sense so that I can get future dependencies in
seconds instead of hours. But I'm fast approaching the point of diminishing
returns so this is my last attempt to ask what is wrong. If I don't hear
from you, I will simply go back to manual methods (or search for some other
tool/task that will find dependencies) and give up on classfileset.

Thanks for your attempt to help me with this.

Rhino

 

[John C. Bollinger]

However, with the original formulation of the classfileset (which you never
criticized) was:

<classfileset id="reqdClasses" dir="${bin.dir}">
<root classname="ca.tonge.exerciseTimer.ExerciseTimer"/>

</classfileset>

Yes, that is fine, supposing always that all the required class files
are arranged suitably relative to ${bin.dir}, and that there are no
required classes that are hidden from dependency analysis (e.g. loaded
only reflectively).

I'm also far from happy that I had to copy the common code under the
ExerciseTimer directory to get this far; I do NOT want to have to do that
routinely if I can possibly avoid it. I have a separate directory under
workspace for each of my projects; I don't want to have to have a single big
project that contains all of the code I have ever written just to enable
classfileset to find the dependencies. But that is the way I interpreted
what you said and I obviously got a better result than I did earlier.

If you're still following this thread, have I misunderstood you yet again?

Not exactly, but it seems that you are still missing at least one key
concept: you do not (in general) need to compile Java classes to the
same directories in which their sources reside. In fact, it quickly
becomes inconvenient to do so in a project of any size. You should not
need to move any sources at any time, but you _should_ compile all the
sources from all your projects to a common destination directory
(without any project-specific subdivisions) in order to enable
ClassFileSet's dependency analysis. Alternatively, you could write an
Ant target to copy all the class files from wherever they are to a
suitable tree, but I'd consider that a kludge.


John Bollinger
(e-mail address removed)

 

[Rhino]

Yes, that is fine, supposing always that all the required class files
are arranged suitably relative to ${bin.dir},

I've described the organization of my files at some length - and even
provided JPGs of the organization ;-) - so can you tell me if my code is
organized "suitably" by your definition? I have a feeling you're one of
those people who can understand the exact application of a rule thoroughly
from only having the principles given to them but I tend to find words a
little slippery so I use concrete examples to make sure that I've understood
the 'rules' correctly....

and that there are no
required classes that are hidden from dependency analysis (e.g. loaded
only reflectively).

That reminds me of something I've been meaning to ask. My Foo class uses
various resource bundles so that it can be multilingual via the standard
techniques developed by the I18N experts at Java. I load these resource
bundles in the standard Java way with the getResource() methods. These
classes and .properties files are in my Foo/Resources package/directory but
have never been picked up by any recursion I've done. Your remark about
"loaded only reflectively" has me wondering just how classfileset works.
Based on my experiences with classfileset so far, I am strongly inclined to
suspect that it only looks at imports and has no way of recognizing resource
bundles as dependents because it doesn't consider any dependent that is
accessed in any way other than via an import statement.

It wouldn't be unreasonable if classfileset can only determine depencies via
imports but it raises the question of whether there are other tools or
techniques to determine *all* dependenies, including resource bundles and
even flat files that my program may read?

again?

Not exactly, but it seems that you are still missing at least one key
concept: you do not (in general) need to compile Java classes to the
same directories in which their sources reside. In fact, it quickly
becomes inconvenient to do so in a project of any size. You should not
need to move any sources at any time, but you _should_ compile all the
sources from all your projects to a common destination directory
(without any project-specific subdivisions) in order to enable
ClassFileSet's dependency analysis. Alternatively, you could write an
Ant target to copy all the class files from wherever they are to a
suitable tree, but I'd consider that a kludge.

Now THAT is an unexpected revelation! It had never seriously occurred to me
to compile all code into the same directory without any project-specific
subdivisions.

I can see where that would probably help classfileset work wonderfully; in
that case, I shouldn't get any of these scope-of-recursion problems at all.
But is that truly "standard practice" to put all code from every project
into the same 'bin' hierarchy?

I just took a look at Eclipse and it is certainly possible to change where
compiled code goes; there is a button to click that draws you a nice tree to
let you select the placement of the compiled code. But the tree doesn't go
any higher than the project level. In my case, that means I can put my
compiled code anywhere within the 'Foo' branch of workspace but not directly
at the workspace level itself. That suggests to me that the gurus at Eclipse
do NOT envision a workspace level 'bin' at all but think that compiled code
should always be segregated at the project level.

So, assuming that neither of you is wrong, does that mean that there should
never be anything like my "Common" project but that "Common" code should
always be within the individual projects where it is actually used?

I'm not prepared to reject that out of hand but I have some serious
reservations about it. Suppose I've removed the classes from my Common
project from my IDE altogether and I store them somewhere else in the file
system so that I can get them when I need them. Let's say one of my classes
from the former Common project handles various date manipulations. I need
this DateManip class for both my Foo and Bar projects because they work with
dates. I copy DateManip (or maybe the entire package containing it) into
both projects Foo and Bar. Now, it turns out that I need to modify DateManip
to suit the needs of projects Foo and Bar. Project Foo needs a new method
called dateFoo() and project Bar needs a new method called dateBar(). I
modify the copy of DateManip in project Foo to add dateFoo(). I modify the
copy of DateManip in project Bar to add dateBar(). All my code works but I
now have three versions of DateManip, the original one that is stored
outside of the the IDE, the version containing dateFoo() in the Foo project
and the version containing dateBar() in the Bar project.

I'm sure you see where this is going. I immediately have issues with regards
to versioning; reconciling these different versions is a serious issue. I
have to be able to distinguish between the versions and/or reconcile them.
I've got distribution issues too: how do I make sure I'm distributing the
right version of DateManip with each project? What happens if I have to make
a change to one of the older methods in DateManip due to a previously
undiscovered bug? Etc. etc.

How do I handle all of these issues? At the moment, I'm developing on my own
and don't need to worry about umpteen developers sharing and changing code
so that keeps things reasonably simple. But I can easily see that even in my
little world, serious complications can result. That's why I strongly prefer
to have common code in its own project that is visible to the other
projects; it means I only need ONE copy of each common module, not one per
project.

What am I missing here?

Rhino

 

[John C. Bollinger]

That reminds me of something I've been meaning to ask. My Foo class uses
various resource bundles so that it can be multilingual via the standard
techniques developed by the I18N experts at Java. I load these resource
bundles in the standard Java way with the getResource() methods. These
classes and .properties files are in my Foo/Resources package/directory but
have never been picked up by any recursion I've done. Your remark about
"loaded only reflectively" has me wondering just how classfileset works.
Based on my experiences with classfileset so far, I am strongly inclined to
suspect that it only looks at imports and has no way of recognizing resource
bundles as dependents because it doesn't consider any dependent that is
accessed in any way other than via an import statement.

ClassFileSet doesn't know anything at all about import statements,
because these have no representation whatsoever in class files -- they
exist only in Java source code. Class files do, however, specify the
fully-qualified names of all other classes on which they directly rely
(superclass, implemented interfaces, declared argument types, declared
return types, declared field types, and declared local variable types).
You probably don't want to know the details, and you don't need to.
ClassFileSet examines each root class and compiles a set of all these
fully qualified names, then recurses to each one (that it can find
within the directory subtree it is working in). You could call this a
static dependency analysis.

It will not find classes loaded only reflectively, a category that
generally includes resource bundle classes. If you need to include such
classes then you need to add them to your list of root classes. It does
not even look for files that are not class files (it is a _CLASS_FileSet).

It wouldn't be unreasonable if classfileset can only determine depencies via
imports but it raises the question of whether there are other tools or
techniques to determine *all* dependenies, including resource bundles and
even flat files that my program may read?

There are no tools that do this, as it is impossible in the general
case. There might be tools that exploit knowledge of the platform API
to find more dependencies than ClassFileSet does, but it is a difficult
proposition with only a modest payoff at best. Better to address the
problem through documentation and suitable development practices.

Not exactly, but it seems that you are still missing at least one key
concept: you do not (in general) need to compile Java classes to the
same directories in which their sources reside. [...]

Now THAT is an unexpected revelation! It had never seriously occurred to me
to compile all code into the same directory without any project-specific
subdivisions.

I can see where that would probably help classfileset work wonderfully; in
that case, I shouldn't get any of these scope-of-recursion problems at all.
But is that truly "standard practice" to put all code from every project
into the same 'bin' hierarchy?

If the projects are independent, then no. But yours aren't independent.

I just took a look at Eclipse and it is certainly possible to change where
compiled code goes; there is a button to click that draws you a nice tree to
let you select the placement of the compiled code. But the tree doesn't go
any higher than the project level. In my case, that means I can put my
compiled code anywhere within the 'Foo' branch of workspace but not directly
at the workspace level itself. That suggests to me that the gurus at Eclipse
do NOT envision a workspace level 'bin' at all but think that compiled code
should always be segregated at the project level.

The gurus at Eclipse can't think of everything. In particular, they did
not anticipate much desire for sort-of-but-not-quite-independent
projects, which is what your "Common" project is. Every one of your
other projects has a different view of "Common", or at least, you want
them all to have, given that this all started with the question of
whether you should distribute the whole thing or just parts of it. You
are caught between conflicting priorities.

If you want ClassFileSet to work for you then you need to get all the
required classes into a common, consistent directory, as we have already
covered. If you insist on using Eclipse for project management (not
necessarily a bad thing) then to make ClassFileSet to work you'll need
to have ant copy around some class files before constructing it.

So, assuming that neither of you is wrong, does that mean that there should
never be anything like my "Common" project but that "Common" code should
always be within the individual projects where it is actually used?

Eclipse's "project" paradigm is apparently not consistent with your
intent for "Common". Eclipse's is not the only paradigm available. You
will recall that I originally recommended that you package "Common" in
several smaller chunks and use those as its units of distribution, and I
still recommend that for common utility classes that are in no way
specific to any particular application. If you have a group of related
applications, however, then it makes some sense to develop them inside a
common code context. That does not necessarily mean a single source
tree, mind you: good build tools, Eclipse included, support multiple
source trees for a single "project".

I'm not prepared to reject that out of hand but I have some serious
reservations about it. Suppose I've removed the classes from my Common
project from my IDE altogether and I store them somewhere else in the file
system so that I can get them when I need them. Let's say one of my classes
from the former Common project handles various date manipulations. I need
this DateManip class for both my Foo and Bar projects because they work with
dates. I copy DateManip (or maybe the entire package containing it) into
both projects Foo and Bar. Now, it turns out that I need to modify DateManip
to suit the needs of projects Foo and Bar. Project Foo needs a new method
called dateFoo() and project Bar needs a new method called dateBar(). I
modify the copy of DateManip in project Foo to add dateFoo(). I modify the
copy of DateManip in project Bar to add dateBar(). All my code works but I
now have three versions of DateManip, the original one that is stored
outside of the the IDE, the version containing dateFoo() in the Foo project
and the version containing dateBar() in the Bar project.

You should not be keeping multiple copies of your sources, except for
versioning purposes. If you have independent, common classes then
distribute the _class files_ as needed.

 

[Rhino]

That reminds me of something I've been meaning to ask. My Foo class uses
various resource bundles so that it can be multilingual via the standard
techniques developed by the I18N experts at Java. I load these resource
bundles in the standard Java way with the getResource() methods. These
classes and .properties files are in my Foo/Resources package/directory but
have never been picked up by any recursion I've done. Your remark about
"loaded only reflectively" has me wondering just how classfileset works.
Based on my experiences with classfileset so far, I am strongly inclined to
suspect that it only looks at imports and has no way of recognizing resource
bundles as dependents because it doesn't consider any dependent that is
accessed in any way other than via an import statement.

ClassFileSet doesn't know anything at all about import statements,
because these have no representation whatsoever in class files -- they
exist only in Java source code. Class files do, however, specify the
fully-qualified names of all other classes on which they directly rely
(superclass, implemented interfaces, declared argument types, declared
return types, declared field types, and declared local variable types).
You probably don't want to know the details, and you don't need to.
ClassFileSet examines each root class and compiles a set of all these
fully qualified names, then recurses to each one (that it can find
within the directory subtree it is working in). You could call this a
static dependency analysis.

It will not find classes loaded only reflectively, a category that
generally includes resource bundle classes. If you need to include such
classes then you need to add them to your list of root classes. It does
not even look for files that are not class files (it is a _CLASS_FileSet).

It wouldn't be unreasonable if classfileset can only determine depencies via
imports but it raises the question of whether there are other tools or
techniques to determine *all* dependenies, including resource bundles and
even flat files that my program may read?

There are no tools that do this, as it is impossible in the general
case. There might be tools that exploit knowledge of the platform API
to find more dependencies than ClassFileSet does, but it is a difficult
proposition with only a modest payoff at best. Better to address the
problem through documentation and suitable development practices.

Not exactly, but it seems that you are still missing at least one key
concept: you do not (in general) need to compile Java classes to the
same directories in which their sources reside. [...]

Now THAT is an unexpected revelation! It had never seriously occurred to me
to compile all code into the same directory without any project-specific
subdivisions.

I can see where that would probably help classfileset work wonderfully; in
that case, I shouldn't get any of these scope-of-recursion problems at all.
But is that truly "standard practice" to put all code from every project
into the same 'bin' hierarchy?

If the projects are independent, then no. But yours aren't independent.

I just took a look at Eclipse and it is certainly possible to change where
compiled code goes; there is a button to click that draws you a nice tree to
let you select the placement of the compiled code. But the tree doesn't go
any higher than the project level. In my case, that means I can put my
compiled code anywhere within the 'Foo' branch of workspace but not directly
at the workspace level itself. That suggests to me that the gurus at Eclipse
do NOT envision a workspace level 'bin' at all but think that compiled code
should always be segregated at the project level.

The gurus at Eclipse can't think of everything. In particular, they did
not anticipate much desire for sort-of-but-not-quite-independent
projects, which is what your "Common" project is. Every one of your
other projects has a different view of "Common", or at least, you want
them all to have, given that this all started with the question of
whether you should distribute the whole thing or just parts of it. You
are caught between conflicting priorities.

If you want ClassFileSet to work for you then you need to get all the
required classes into a common, consistent directory, as we have already
covered. If you insist on using Eclipse for project management (not
necessarily a bad thing) then to make ClassFileSet to work you'll need
to have ant copy around some class files before constructing it.

So, assuming that neither of you is wrong, does that mean that there should
never be anything like my "Common" project but that "Common" code should
always be within the individual projects where it is actually used?

Eclipse's "project" paradigm is apparently not consistent with your
intent for "Common". Eclipse's is not the only paradigm available. You
will recall that I originally recommended that you package "Common" in
several smaller chunks and use those as its units of distribution, and I
still recommend that for common utility classes that are in no way
specific to any particular application. If you have a group of related
applications, however, then it makes some sense to develop them inside a
common code context. That does not necessarily mean a single source
tree, mind you: good build tools, Eclipse included, support multiple
source trees for a single "project".

I'm not prepared to reject that out of hand but I have some serious
reservations about it. Suppose I've removed the classes from my Common
project from my IDE altogether and I store them somewhere else in the file
system so that I can get them when I need them. Let's say one of my classes
from the former Common project handles various date manipulations. I need
this DateManip class for both my Foo and Bar projects because they work with
dates. I copy DateManip (or maybe the entire package containing it) into
both projects Foo and Bar. Now, it turns out that I need to modify DateManip
to suit the needs of projects Foo and Bar. Project Foo needs a new method
called dateFoo() and project Bar needs a new method called dateBar(). I
modify the copy of DateManip in project Foo to add dateFoo(). I modify the
copy of DateManip in project Bar to add dateBar(). All my code works but I
now have three versions of DateManip, the original one that is stored
outside of the the IDE, the version containing dateFoo() in the Foo project
and the version containing dateBar() in the Bar project.

You should not be keeping multiple copies of your sources, except for
versioning purposes. If you have independent, common classes then
distribute the _class files_ as needed.

Ok, I guess we've pretty much beaten this to death. I see now why I wasn't
having much luck with my classfileset; I simply had my files organized
inappropriately. It just took us a while to figure that out ;-)

Thanks for your help with this matter.

I need to mull over the best way to organize my code now ;-)

Rhino