A Large-Scale Empirical Study of the Relationship between Build Technology and Build Maintenance

Abstract

Build systems specify how source code is translated into deliverables. They require continual maintenance as the system they build evolves. This build maintenance can become so burdensome that projects switch build technologies, potentially having to rewrite thousands of lines of build code. We aim to understand the prevalence of different build technologies and the relationship between build technology and build maintenance by analyzing version histories in a corpus of 177,039 repositories spread across four software forges, three software ecosystems, and four large-scale projects. We study low-level, abstraction-based, and framework-driven build technologies, as well as tools that automatically manage external dependencies. We find that modern, framework-driven build technologies need to be maintained more often and these build changes are more tightly coupled with the source code than low-level or abstraction-based ones. However, build technology migrations tend to coincide with a shift of build maintenance work to a build-focused team, deferring the cost of build maintenance to them.

Appendices

Appendix

1.1 A Build Technology Examples

In this appendix, we briefly describe how each of the studied technologies can be used to specify a simple build system.

1.2 A.1 Low-Level

Figure 17 provides working examples of the five studied low-level build technologies.

Fig. 17

Example low-level technology specifications

Make

One of the earliest build technologies on record is Feldman’s make tool (Feldman 1979), which automatically synchronizes program sources with deliverables. Make specifications outline target-dependency-recipe tuples. Targets specify files created by a recipe, i.e., a shell script that is executed when the target either: (1) does not exist, or (2) is older than one or more of its dependencies, i.e., a list of other files and targets.

The make specification snippet in Fig. 17a describes three target-dependency-recipe tuples. Lines 2, 4, and 7 list targets to the left of the colons and dependency lists to the right. Recipes are specified for the main.o and example targets on lines 5 and 8. Line 1 of Fig. 17a specifies that the all target is phony, representing an abstract phase in the build process rather than a concrete file in the filesystem.

Jam

Jam provides a more procedural-style structure for target-dependency-recipe tuples. Figure 17b shows how rules (the equivalent of make tuples) can be specified (lines 1–4 and 10–13). Dependencies are expressed by invoking the built-in Depends rule on lines 2 and 11. Jam actions (the equivalent of make recipes) for C compilation and object code linking are defined on lines 6–8 and 15–17 respectively.

Ant

Ant borrows the target-dependency-recipe concept from make, however all Ant targets are abstract. When an Ant target is triggered, a list of specified tasks (the equivalent of make recipes) are invoked. Ant tasks execute Java code rather than shell scripts to synchronize sources with deliverables.

Figure 17c shows an Ant specification that describes two targets, i.e., compile (lines 2–8) and link (lines 10–18). The compile target invokes the javac task (lines 3–7), which executes the javac compiler. The link target invokes the jar task (lines 14–17), which executes the jar command. The dependency between the link and compile targets is expressed on line 12 using the depends target attribute.

SCons

SCons provides several advanced build system features (e.g., implicit dependency tracking for popular programming languages) and allows maintainers to write highly portable build specifications using Python. Line 7 of Fig. 17d shows how a binary example can be assembled from object code. Line 5 shows how object code can be generated using SCons built-in support for C ++ compilation. Environmental settings (e.g., compilers, linkers, and flags) are automatically detected, however parameters passed to the Environment() function call will override the detected settings, as shown on line 1.

Rake

Rake is a modern build tool with advanced support for building Ruby applications. Similar to SCons, Rake specifications are written in a high-level scripting language (i.e., Ruby), to give build maintainers the power to express complex relationships and transformations in a highly portable language. Similar to Ant, Rake tasks (the equivalent of targets in make) are abstract.

The example snippet in Fig. 17e shows how a unit testing task utest can be specified (lines 3–5). Line 4 describes the recipe that is executed when utest is triggered. Line 1 specifies that the default target depends upon the utest target.

1.3 A.2 Abstraction-Based

Figure 18 provides working examples of the two studied abstraction-based technologies.

Fig. 18

Example abstraction-based technology specifications

Autotools

GNU Autotools specifications describe external and internal dependencies, configurable compile-time features, and platform requirements. These specifications are parsed to generate make specifications that satisfy the described constraints.

Autotools is actually a large collection of build tools that work together to generate build systems according to specifications. Two of the most commonly used tools are autoconf and automake, for which we provide example specifications in Fig. 18a and b respectively. Lines 1 and 2 of Fig. 18a initialize the autoconf environment, specifying that our project name is example version 1.0 and that automake is also necessary. Line 3 specifies an environment dependency on a C compiler, while lines 4 and 5 request that the configuration step store preprocessor directives in a file named config.h, and store the build system implementation in a file called Makefile. Line 1 of Fig. 18b specifies that a deliverable called example should be constructed during the build process and that it should be deployed in the bin directory. Line 2 states that main.c is a source file that should be compiled and linked into the example binary.

CMake

Similar to Autotools, CMake abstractions can be used to generate make specifications, but can also generate Microsoft Visual Studio and Apple Xcode project files. Figure 18c specifies that a build system should be generated to produce a binary called example by compiling and linking main.cc (line 4) as a part of a project called Example (line 2). Line 1 denotes that CMake version 2.6 (or later) should be used to parse the specification.

1.4 A.3 Framework-Driven

Below we describe the studied Maven framework-driven technology.

Maven

Maven assumes that source and test files are placed in default locations and that projects adhere to a typical Java dependency policy, unless otherwise specified. If projects abide by the conventions, Maven can infer build behaviour automatically without any explicit specification. For example, Fig. 19a does not specify a location for source or output files. Convention specifies that source and unit test code appear under src/main/java and src/test/java respectively.

Fig. 19

Example Framework-driven and dependency management technology specifications

Lines 10–18 of Fig. 19a show how the Maven convention can be overridden through configuration. The Java compiler is instructed to operate in Java 1.5 source mode (line 15), and generate bytecode that is compatible with the Java 1.7 runtime environment (line 16).

1.5 A.4 Dependency Management

Figure 19 provides working examples of dependency management in Maven (Fig. 19a) and the two studied dependency management technologies (Fig. 19b and c).

Maven

In addition to providing a framework-driven build environment, Maven doubles as a dependency management technology. Lines 22–26 of Fig. 19a provide an example dependency declaration on the JUnit tool, version 3.8.1 (Figs. 20 and 21).

Fig. 20

Monthly build commit proportion, sizes, and churn volume in the studied forges

Fig. 21

Monthly source-build coupling and build author ratios for the studied forges

Ivy

Ivy provides dependency management features that are most notably leveraged by Ant. Figure 19b shows an Ivy specification for the same JUnit dependency as depicted in Fig. 19a.

Bundler

Bundler provides packaging and dependency management for Ruby applications. Line 1 of Fig. 19c specifies that bundler should download gems, i.e., Ruby packages, from the given host. Lines 2 and 3 specify dependencies on Rake version 10.0.3 (at least) and rspec version 2.13.0 (exact).

B Additional Build Maintenance Figures

We perform longitudinal analyses of the Tukey HSD ranks for each metric in the forges to complement our median-based analyses in Section 6. Figures 20 and 21 show only the first twelve months of history and the top three ranks to improve the readability of the figures. Unfiltered figures are available online.¹⁵