javac vs Makefile problem ...

[for.fun]

Hi everybody,


I actually have the following problem :

I have a Java/JNI application which mixes ".h", ".cpp", ".java" (native
and regular files)
In order to compile everything easily, I did a Makefile.

I know that "javac" resolves the dependencies but because I have to
generate ".h" files from ".class", I had to include the Java
compilation in my Makefile.

To be clearer, here is my compilation chain :

javac javah CC
X.java => X.class => X.h => X.o
X.cpp


All this chain is achieved thanks to my Makefile.
It works but not as well as I expect it to.


My problem is :

1/ Consider that "Y.java" depends on "X.java"

2/ If "Y.java" is compiled later, it involves "X.java" to be compiled
again.

3/ Consequently, the modify time of "X.java" is changed so all the long
job starting from "X.class" is done again and my Makefile takes a while
....


Is there a way to disable the "javac" automatic dependency resolution
in order to completely manage it in a Makefile ?
Do you have another issue which could help me ?


Thanks in advance.

 

[Adam Maass]

Hi everybody,


I actually have the following problem :

I have a Java/JNI application which mixes ".h", ".cpp", ".java" (native
and regular files)
In order to compile everything easily, I did a Makefile.

I know that "javac" resolves the dependencies but because I have to
generate ".h" files from ".class", I had to include the Java
compilation in my Makefile.

To be clearer, here is my compilation chain :

javac javah CC
X.java => X.class => X.h => X.o
X.cpp


All this chain is achieved thanks to my Makefile.
It works but not as well as I expect it to.


My problem is :

1/ Consider that "Y.java" depends on "X.java"

2/ If "Y.java" is compiled later, it involves "X.java" to be compiled
again.

3/ Consequently, the modify time of "X.java" is changed so all the long
job starting from "X.class" is done again and my Makefile takes a while
...


Is there a way to disable the "javac" automatic dependency resolution
in order to completely manage it in a Makefile ?
Do you have another issue which could help me ?

There is no way to disable the javac automatic dependency resolution.

However, there is a way to fix your build script so that it ends up doing
what you want:

You don't want 1 execution of javac for each .java file in your project;
instead, what you want to do is invoke javac once on *all* .java files in
your project -- all at one go. Do this first, then do the javah and
subsequent steps.

If all of your .java files live in one directory (or are contained in
subdirectories of one directory), your life is much simplified. Examine
the -sourcepath option to javac.

 

[Roedy Green]

Is there a way to disable the "javac" automatic dependency resolution
in order to completely manage it in a Makefile ?
Do you have another issue which could help me ?

Almost no one use a makefile anymore. They use ANT to handle this,
which solves the problem in a platform-independent way.

You periodically do a clean compile (recompile the universe). Other
than than that, A depending on B won't recompile B. If B has changed
then A might get recompiled, but there are situations where A should
be recompiled, but is not. I don't know of any situation where you
get spurious recompiles.

The more common problem is failing to rebuild all jars that contain A
when A changes.

see http://mindprod.com/jgloss/ant.html

 

[Roedy Green]

You don't want 1 execution of javac for each .java file in your project;
instead, what you want to do is invoke javac once on *all* .java files in
your project -- all at one go. Do this first, then do the javah and
subsequent steps.

ANT is clever enough to invoke JavaC only once no matter how
complicated your build script. It is at least a order of magnitude
faster than invoking Javac for each *.java file.

 

[Jeffrey Schwab]

There is no way to disable the javac automatic dependency resolution.

However, there is a way to fix your build script so that it ends up doing
what you want:

You don't want 1 execution of javac for each .java file in your project;
instead, what you want to do is invoke javac once on *all* .java files in
your project -- all at one go. Do this first, then do the javah and
subsequent steps.

If all of your .java files live in one directory (or are contained in
subdirectories of one directory), your life is much simplified. Examine
the -sourcepath option to javac.

OP: Of course Adam's solution is probably optimal. Just out of
curiousity, why is the modification time of X.java changing? Compiling
a source file should not change the source's modification time.

 

[for.fun]

Jeffrey Schwab a écrit :

OP: Of course Adam's solution is probably optimal. Just out of
curiousity, why is the modification time of X.java changing? Compiling
a source file should not change the source's modification time.

Of course, you are right : the source Java modify time will not change.
I did a write mistake.
In facts, I meant this : 'the modify time of "X.class" is changed' and
so on ...

I had a quick look to ANT.
I understand your point of view but I am not going to use it because I
think that the Unix shell commands are powerfull and I can not work
without them.
Moreover, it is no problem installing Win32 ports of the Unix shell
commands (direct port or using Cygwin) on any system so you can run
make everywhere.
I just did it did and it works fine on Solaris and Windows XP.

The Adam Maass will work but I will not gain any performance : each
time I will touch a Java source file, everything is recompiled. I work
on a very big project and recompiling anything each time will take
hours ...

 

[for.fun]

Today, I spent some time looking for some other Java tools and I
discovered the Jikes Java compiler which is said to be compatible with
the JVM.
Moreover, Jikes allows to manually manage dependencies by generating
dependencies Makefile.

I am going to try this ...


Adam Maass a écrit :

 

[Adam Maass]

Almost no one use a makefile anymore. They use ANT [instead].

Not true. My current project is a combination of Ant and make; Ant exec's
make for the native portions of the project. There's good reason for
makefiles if there's no portion of the project that's Java.

If the project is pure Java (or nearly so), then I can see using Ant as
*the* build executible. Otherwise, makefiles have a lot going for them.



-- Adam Maass

 

[Adam Maass]

To be clearer, here is my compilation chain :

javac javah CC
X.java => X.class => X.h => X.o
X.cpp

I suggested compiling all of the .java files to .class files at one go,
using the -sourcepath argument to javac.

The Adam Maass will work but I will not gain any performance : each
time I will touch a Java source file, everything is recompiled. I work
on a very big project and recompiling anything each time will take
hours ...

OP: try setting the dependency your X.h and X.cpp files not on X.class, but
X.java instead. That should alleviate the "recompile the world when anything
changes" problem.

-- Adam Maass

 

[for.fun]

Adam Maass a écrit :

OP: try setting the dependency your X.h and X.cpp files not on X.class, but
X.java instead. That should alleviate the "recompile the world when anything
changes" problem.

Good idea. I am going to try it.
Thx Adam.

 

[for.fun]

Adam Maass a écrit :

OP: try setting the dependency your X.h and X.cpp files not on X.class, but
X.java instead. That should alleviate the "recompile the world when anything
changes" problem.

I tried your issue but it leads to another problem.
When I add the dependency "%.h : %.java", "make" must know how to make
".h" from ".java".

To make ".h" from ".java", you must :

1/ make the ".class" with "javac"
2/ make the ".h" with "javah"

As you said, the best way to do it is to compile all the world with
"javac".
With your issue, I have to recompile all the world with "javac" each
time I have a ".h" to generate.


I found an issue which works even if I am not self-satisfied !
Anyway, I gained a lot of time doing like this :

Instead of relying on built-in "make" processus based on modified file
times, I implemented a CRC-based processus.
I the Java class CRC change after a "javac" then I make the ".h"
otherwise I do nothing.

This works fine because I noticed that most of time "javac" regenerates
the same classes binaries.

"javac" is not really an optimized compiler !

=> Why recompiling "n" times the same classes ?

 

[Gordon Beaton]

I tried your issue but it leads to another problem. When I add the
dependency "%.h : %.java", "make" must know how to make ".h" from
".java".

But you don't make the header from the java source, you make it from
the class. In fact that's what you say here:

To make ".h" from ".java", you must :

1/ make the ".class" with "javac"
2/ make the ".h" with "javah"

The dependency rule should be: "%.h: %.class" instead.

/gordon

 

[for.fun]

Gordon Beaton a écrit :

On 5 Dec 2005 08:39:23 -0800, (e-mail address removed) wrote:

The dependency rule should be: "%.h: %.class" instead.

Yes, I know but my first problem was due to "javac" which recompiles
any Java class even if the Java source did not change. Indeed, all
headers are recompiled and nearly all my project was rebuilt.
I work on a huge project so rebuilding everything take a while and the
purpose of "make" was to avoid that kind of situation.

That is why Adam suggested me to depend on ".java" instead of ".class"
in order to avoid the full rebuild.

I am not really happy with "javac" which works for nothing !

 

[Thomas Weidenfeller]

Yes, I know but my first problem was due to "javac" which recompiles
any Java class even if the Java source did not change.

You should really check if you don't have a second version of (older)
..class files in the classpath.

javac uses the directory as specified with -d to store compiled .class
files, however, it does not automatically search in that directory when
looking for type information. Instead, it uses the classpath, and, when
defined, the sourcepath to search for classes and source code. If the
directory specified with -d is not in the classpath (having it there
first is a very good idea), then you are in trouble. Particular, because
the default for the classpath is the current directory.

So, if you happen to have old .class files in the classpath (current
directory) first, javac thinks the corresponding source needs
recompilation. But instead of replacing the old *.class files, it writes
the compilation result to the -d directory. The old *.class files remain
untouched in this case, and next time you compile javac once again
thinks it needs to recompile them.

In make parlance, using something like

OUTPUT_DIR = classes
CLASSPATH = $(OUTPUT_DIR):<rest of classpath>
JFLAGS = -d $(OUTPUT_DIR) -cp $(CLASSPATH) <more options>

etc. is a very good idea.

Indeed, all
headers are recompiled and nearly all my project was rebuilt.
I work on a huge project so rebuilding everything take a while and the
purpose of "make" was to avoid that kind of situation.

make is still a fine tool, and it is very much possible to use it to
build java applications. make's biggest problem with java is that most
versions of make don't allow to specify cyclic dependencies.
Unfortunately, these are common in java, where class A refers to class
B, and class B to class A. If you have such a normal make, you need to
group java source files into sets, where each set only has cyclic
dependencies within the set, and each set is compiled with a single
invocation of javac. The dependencies among sets should form a DAG,
which is what make expects for dependencies.

Some time ago I considered to write a special make for Java, which
allows to specify cyclic dependencies, and does the grouping into sets
of cyclic dependent source code internally. However, the atrocity called
ant was already gaining rapid popularity, and I didn't see any change of
widespread usage of such a special make.

/Thomas

 

[for.fun]

Thomas Weidenfeller a écrit :

First of all, thanks for all your detailed explanations.

In make parlance, using something like

OUTPUT_DIR = classes
CLASSPATH = $(OUTPUT_DIR):<rest of classpath>
JFLAGS = -d $(OUTPUT_DIR) -cp $(CLASSPATH) <more options>

etc. is a very good idea.

In facts, that is what I do in my Makefile, I generate the ".class" in
an output directory (specified by "-d") and include that dir into my
classpath. I double-checked and I am sure that I have no older version
of my ".class" in my directories.


Here is an example of what happens to me (in the make context) :

1/ I have the class A which depends on B, C, D

2/ When A change, "make" will call "javac" which will recompile A
Unfornutately, "javac" also recompile B, C, D.

3/ Consequently, the JNI headers are rebuilt for A but also for B, C, D
while it was not necessary.

make is still a fine tool, and it is very much possible to use it to
build java applications. make's biggest problem with java is that most
versions of make don't allow to specify cyclic dependencies.

I did not know that. I think GNU make does not allow cyclic
dependencies because I can see plenty of "Avoiding implicit rule
recursion." in the verbose make logs.
.... and I suppose that a cyclic dependency can be interpreted as a
recursion by GNU make.

Some time ago I considered to write a special make for Java, which
allows to specify cyclic dependencies, and does the grouping into sets
of cyclic dependent source code internally. However, the atrocity called
ant was already gaining rapid popularity, and I didn't see any change of
widespread usage of such a special make.

I was advised to use Ant by people on this forum but after a quick
look, it does not seem as powerful as make (but I may be wrong)

And I like "make" too much (especially "GNU make")

 

[Thomas Weidenfeller]

Thomas Weidenfeller a écrit :

First of all, thanks for all your detailed explanations.





In facts, that is what I do in my Makefile, I generate the ".class" in
an output directory (specified by "-d") and include that dir into my
classpath. I double-checked and I am sure that I have no older version
of my ".class" in my directories.


Here is an example of what happens to me (in the make context) :

1/ I have the class A which depends on B, C, D

2/ When A change, "make" will call "javac" which will recompile A
Unfornutately, "javac" also recompile B, C, D.

It only should do this when the found class files are older than the
source file, or it couldn't find the class files at all. Check your
system time, check the time on the files, and you could also check which
files javac evaluates by e.g. running javac with the -verbose option. If
you don't trust javac's verbose output you could also run it under the
control of truss ("truss -f -t open -t stat javac ..." should do) to see
what it really does.

I did not know that. I think GNU make does not allow cyclic
dependencies because I can see plenty of "Avoiding implicit rule
recursion."

Oh no, that indicates that something is wrong with the way you have set
up your implicit rules. It does not indicate that you declared a
circular dependency among source files. Something in your pattern rules,
your suffix rules, or the .SUFFIXES list is broken, seriously broken.

That would also be a possible explanation for your continuous
recompilation: You simple provided make with a broken rule set.

Circular dependencies in GNU make should give you a warning like
"circular <some dependency> dropped". Other makes simply stop with an
error message. But even if you use GNU make, you should fix the Makefile
to get rid of the circular dependency specifications, and instead group
your source in the previously mentioned sets.

/Thomas

 

[for.fun]

Thomas Weidenfeller a écrit :

That would also be a possible explanation for your continuous
recompilation: You simple provided make with a broken rule set.

You are probably right.
I am going to look this way.

Thanks a lot for all your usefull advices.