COMP2100/2500 Software Construction -- 2009

Lecture 13: Build Automation. Make and Ant

Summary

An introduction to two of the standard tools for managing the build process of large programs: the classical Make and the modern Java specific Ant.

Aims

Build Tools

Why do we need build tools?

What can a build tool do for you?

Why a Build Tool is important?

Can't smart compilers like the one for Java do that for me?

Yes, but:

File Dependency Example

Converter.java

Source file for Converter class

"other classes"

Compiled Java classes used by Converter

oops

a bash script for running Converter through JVM

sample.sxw

Test (sample) OO document file

sample.html

Output html-file of running sample.sxw through oops

sample.html

Output html-file of running sample.sxw through oops

docs (phony)

Documentation html-files generated by javadoc tool

oops.jar

Java archive (JAR) file

Dependency diagram for the oops system

How to use a Build Tool?

You must tell a build tool the following things about your system:

Operating systems tag files with the time of their last update. Build tools compare the ages of files and rebuild new versions appropriately.

For example, if sample.html is missing or older than oops or sample.sxw, a build tool would run `oops sample.sxw' to update it.

Make: A UNIX Build Tool

The UNIX make command requires an input file, called `makefile' (or `Makefile'), to describe the dependencies between files and how to update them.

A makefile consists of multiple entries of the following format:

file-name: other files it depends on
        commands to build it

This is called a production block. The file named before the colon is called a target. Those which follow the colon are called dependencies. Commands which must be executed if the target is older than at least one of the dependencies, follow the Tab character on the following line. The appearance of this tabbed format is rather fortuitous, as explained by the Make creator Stuart Feldman.

The production blocks are separated by empty lines.

To update the file f with respect to the files on which it depends, type `make f' into the shell.

Just saying `make' updates the first target named in makefile.

An Example Makefile

# Makefile for the Oops system Java implementation
#
# Author: Ian Barnes
# $Revision: 2.3 $
# $Date: 2009/04/02 02:52:25 $

default: test

sources = $(wildcard ./comp2100/oops/*.java ./comp2100/oops/*/*.java)
classes = $(sources:.java=.class)
samples = $(wildcard tests/sample*.sxw)

oops: $(classes) Makefile
javac $(sources)

clean:
        rm -f $(classes)
        rm -f ./comp2100/oops/*~
        rm -f ./comp2100/oops/*/*~
        rm -rf docs/
        rm -f tests/*.txt tests/*.xml tests/*.html

%.class: %.java
        javac -classpath . $<

docs: $(classes) Makefile
        javadoc -d docs -classpath . -author -version \
        -link http://java.sun.com/j2se/1.5.0/docs/api/ \
        -linksource -private -use comp2100.oops.tree \
        comp2100.oops.scanner comp2100.oops.visitor \
        comp2100.oops

%.txt: %.sxw
        ./oops $<

%.xml: %.sxw
        ./oops $<

test: oops $(samples)
        ./run_tests

jar: oops.jar

oops.jar: $(sources)
        @jar cvf $@ $(sources)

Using the example of Makefile

Going once...
partch:~/comp2100/2006/ass/a1> make oops
javac ./comp2100/oops/Assert.java ./comp2100/oops/Converter.java ...
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

Going twice...
partch:~/comp2100/2006/ass/a1> make clean
rm -f ./comp2100/oops/Assert.class ./comp2100/oops/Converter.class ...
rm -f ./comp2100/oops/*~
rm -f ./comp2100/oops/*/*~
rm -rf docs/
rm -f tests/*.txt tests/*.xml tests/*.html

Going thrice...
partch:~/comp2100/2006/ass/a1> make jar
added manifest
adding: comp2100/oops/Assert.java(in = 1483) (out= 535)(deflated 63%)
adding: comp2100/oops/Converter.java(in = 5571) (out= 1508)(deflated 72%)
adding: comp2100/oops/StringOps.java(in = 2008) (out= 790)(deflated 60%)
........

Going ...? for the fourth time...
partch:~/comp2100/2006/ass/a1> make -s test
tests/test.sxw:
This is Oops - the Open Office Publishing System
Creating the scanner which feeds on the input stream
Parsing the input and creating the root element
Removing bad paragraphs
Gathering style information
Style information gathered:
P1 --> Author's Name
..................
Extracting metadata
Metadata information collected:
Title         = "A Short Paper"
Author's Name = "Ian Barnes"
Affiliation   = "Australian National University"
Opening tests/test.txt for plain text output
...............
Finished writing the HTML file
XML OK
TXT OK

Phony Targets

You can declare targets that give names to actions that do not correspond to files. These are called phony targets.

Example: We can use a phony target to remove all the automatically generated files. 
clean:
        rm -f $(classes)
        rm -f ./comp2100/oops/*~
        rm -f ./comp2100/oops/*/*~
        rm -rf docs/
        rm -f tests/*.txt tests/*.xml tests/*.html

If you run make clean, make should discover that there is no file called `clean', so it runs the command. (After which, there is still no file called `clean'.)

We also can use a phony target as a good mnemonics:

jar: oops.jar

The jar does not exist, and will not be created, but it depends on oops.jar, and its "up-to-dateness" will be checked, and if needed it will be rebuild (which is the desired effect).

More Phony Targets

We can use a phony target to build a list of files.

all: file1 file2 ...

If you run make all, make should discover that there is no file called `all'. Before it realises that there is no command to build all, it will first update each of the dependencies.

Note: In most makefiles, the first target is a phony target called all that depends on a list of files to be updated when Make is run without arguments.

Implicit and pattern-matching rules, dynamic macros

Often the dependencies are quite large, and writing down the long list of them is impractical (what? edit the Makefile every time a new source file is added?). This is not necessary. The Makefile allows to include predefined macros, like

sources = $(wildcard ./comp2100/oops/*.java ./comp2100/oops/*/*.java)

to eliminate the need to edit makefiles when the underlying compilation environment changes. Some predefined macros can include the substitution rule, like this

classes = $(sources:.java=.class)

which says "take the whole list of $(sources) and replace the suffix .java onto the suffix .class by thus getting the list of $(classes)".

The Make program can use pattern-matching rules to build the dependency list dynamically (ie, during the Make run). In the Makefile above, the dependency block

%.txt: %.sxw
        ./oops $<

is an example of such rule: the wildcard % here stands for a base name common in both target and dependency names.

(In the language C, this can work even in a case of an absent (implicit) rule. If the pattern-matching has a form:

%.o: %.c

then Make will know that it has to use the rule `gcc -c' applied to the matching name.)

Finally, macros which build dynamically with the implicit rules, change from one production block to another. The production rules in such situation are defined by the use of dynamic macros. The examples from above:

%.class: %.java
        javac -classpath . $<

oops.jar: $(sources)
        @jar cvf $@ $(sources)

These are some often used dynamic macros:

$@

The name of the current target

$?

The list of dependencies newer than the target

$<

The name of the dependency file selected by Make

$*

The base name of the current target

Problems with Make in Java projects

Make is universal, Make is plain test based (portable across platforms, can be generated by programs), and when used on C-based projects, Make can derive dependencies between the source files (like the above mentioned makedepend tool which may be regarded as a part of the Make suite). But for Java...

One has to acknowledge that Makefile script which was discussed above (the very same given for Assignment 1) is an overkill. It recompiles every class every time, whether or not it has changed. In principle, you should now be able to write a Makefile that recompiles the project without waste.

But it's not that simple. To start with, you would need rules expressing the dependency between X.java and X.class (for each class X) so that if you modify X.java it will recompile it. Use of the implicit rules helps to arrest the Makefile "inflation".

But unfortunately it's not enough. If it was a C language project this could be dealt with (explain). But Java is OO language, it has inheritance. Java classes are not all independent. Some of them depend on others. This can be through use (one class is named in a declaration somewhere in the other), or through inheritance (one class extends another class or implements an interface). In order to compile a class correctly, the compiler needs an up-to-date .class file for all the classes it depends on. Now the Java compiler knows a bit about dependencies: suppose class X depends on class Y, and class Y has been changed since the last compilation. If I tell the Java compiler to compile X, it will check whether Y.class is up to date, and will recompile it if it isn't. That's good. But if I ask it to compile class Y, since it is the only one that changed, the Java compiler has no way of knowing that class X depends on it, and should now be recompiled too. This is what our Makefile would do, and it's wrong, and can lead to really weird problems at runtime. This is all a bit of a nightmare, and generally leads to people adopting a conservative build management strategy: recompile everything every time, just in case.

The solution to this seems obvious, and Make was made for it. If class X depends on class Y, we put a line in our Makefile like this:

X.class: Y.class

We don't need the action part (the second line with the instruction indented by a tab), since the implicit rule tells Make how to build X.class if it needs to. All this does is express the information that if Y has changed, we need to recompile X.

All done, right? Wrong. And this is a killer. The problem is that many Java programs have cyclic dependencies between the classes. Look at the Visitor pattern in the project for example. The interface depends on class XmlContainerElement and XmlContainerElement depends on Visitor. If either one of these changes, the other one should be recompiled. You need both dependencies in your Makefile.

Visitor.class: XmlContainerElement.class
XmlContainerElement.class: Visitor.class

The problem is that when Make see that, it gives up. Make works on the model where dependencies form a tree (like the diagrams in the lecture). This is valid for C programs, but not for Java. When Make sees a cyclic dependency, it prints an error message and stops. The result is that we can't write a Makefile for our project.

Another Neat Tool

Apache's Ant comes to the rescue. Ant is a Java-based build tool for Java. Instead of a Makefile, it uses an XML file called build.xml.

To run Ant, you will need to define the following environment variables before it will work correctly.

setenv ANT_HOME /usr/share/ant
setenv JAVA_HOME /usr/local/java

Put those lines in your .cshrc file (if your shell is bash use export command in .bashrc) if you are going to use Ant regularly.

Ant first example

Ant build files have very great differences in syntax from make's makefiles, and the two tools also have very diferent virewpoints. While make starts from productions, rules, and files (and then commands to update the files), Ant starts with a set of tasks which contribute to managing projects.

Like Make, Ant uses instructions written in a plain text file (great for portability, code generation and posterity), but instead of the quirky tabbed format, its scripts are written in XML. The Ant script is normally called build.xml, but (like Make) it can also take its instructions from an arbitrary named file. The Ant script has a typical XML structure: nested tags with attributes.

Let's say I have a small Java project (one file only) with the following source in the file Project.java

public class Project {
        public static void main(String args[]) {
                System.out.println("No worries, mate!");
        }
}

We may compile it, generate javadoc documentations, we can put it into .jar file, etc. All these tasks can be programmed in build.xml script.

<?xml version="1.0" ?>
<project default="main">

         <target name="main" depends="compile, compress">
                <echo>
                        Building the .jar file.        
                </echo>
        </target>

        <target name="compile">
                <javac srcdir="."/>
        </target>

        <target name="compress">
                <jar jarfile="Project.jar" basedir="." includes="*.class" />
        </target>

        <target name="docs">
                <javadoc  sourcefiles="Project.java" destdir="." />
        </target>

</project>

The script can be run like this:

abx@eudyptula:~/comp2100/2006/lectures/lec-make-ant$ ant build.xml
Unable to locate tools.jar. Expected to find it in /usr/lib/sablevm/lib/tools.jar
Buildfile: build.xml

BUILD FAILED
/home/abx/comp2100/2006/lectures/lec-make-ant/build_project.xml:12: The element type "javac" must be terminated by the matching end-tag "".

Total time: 0 seconds

I forgot to set the environment variables (as usual). Doing this ...

export ANT_HOME=`which ant`
export JAVA_HOME=/usr/local/jdk

and trying again:

abx@eudyptula:~/comp2100/2006/lectures/lec-make-ant$ ant 
Buildfile: build.xml

compile:

compress:
     [jar] Building jar: /home/abx/comp2100/2006/lectures/lec-make-ant/Project.jar

main:
     [echo]
     [echo]                         Building the .jar file.
     [echo]

BUILD SUCCESSFUL
Total time: 1 second

Now, I can play and call ant with all the targets present in the build.xml file to achieve the goals which are defined in those targets. As you can see, the build file is a collection of such predefined targets, which are expressed as named tags with (some) attributes, eg name, depends etc, and inside the target tags you can include nested tags which will describe actions to be taken for performing the tasks which constitute the "parental" target. Those tasks will be undertaken only after the tasks in the depends target list are completed first.

Ant has a number (quite a big one) of built-in (core) tasks. See the manual: Overview of Ant tasks. They include all possible actions (well, really only many of the Ant group's\ anticipated actions) needed for managing a Java based project. In different operating systems they will be interpreted in accordance with the local lore. That's why the Ant build scripts are portable accross platforms like the Java projects themselves. Among those built-ins there is ant, ie ant can call ant. You can also define new tasks. Apart from targets, a build script can contain property tags, in which you can set "name=value" pairs similar to macros in the Makefile.

We will practice in writing a build script as an extension task in a lab later in the course.

Unlike Make, Ant can handle the dependency problem using special depend task. The <depend> tag and its attributes (which are normally set with the help of properties) allow you to perform the dependency check (which is the task of the depend target) and find out whether there are out-of-date classes. If it finds them, it removes the .class files of any other classes that depend on them. To determine dependencies, this task analyses the classes in all files passed to it, using the class references encoded into .class files by the compiler (it does not parse or reads the source code).

If you want Ant to be super-conservative, you can turn on the closure attribute of the depend task, and it will delete any .class file that depends on anything that depends on anything that is out-of-date, and so on until it has followed all chains of dependencies to the end.

Additional reading for Make

Additional reading for ANT

Additional examples for Ant

A more elaborate build file will be distributed with the second assignment version of Oops.

Copyright © 2006, 2009, Jim Grundy, Ian Barnes, Richard Walker, Alexei Khorev, Chris Johnson, The Australian National University
$Revision: 2.3 $ $Date: 2009/04/02 02:52:25 $
Feedback & Queries to comp2100@cs.anu.edu.au