Grow your projects

When I enter a new project, I usually start by write a Makefile (or equivalent) to handle compilation of that project. Now, with careful crafting, a standalone Makefile can be just as expressive and powerful as an automatically-generated monstrosity, but it takes lots of pain and efforts to get it to handle dependencies correctly.

Sometimes, despite my best efforts, my Makefiles keep recompiling the wrong things, either too much or too little. Don’t even get me started on the brain-dead way they have decided to implement ‘#include’, where included files are executed twice, once before recompiling and once after, which leads to (filesystem) cache corruption when a build fails.

I am aware of other such tools existing, such as Ant (XML configuration specialized to build Java applications, yuck…), Cons (gets the job done, but too complex for my taste) or Cook (also interesting, but suffers some of Make’s shortcomings in complex dependency graphs). None of these tools really strive for simplicity, however, and many of them fail to scale well on complex builds, often requiring dirty tricks or hierarchical builds to achieve a simple outcome.

Of those tools, only Cons achieves truly correct dependency resolution, but that is more by effort than elegance, as it is much more complex than a build system really needs to be.

Enter Grow

Grow is a simple make-like program that reads a seed file in the current (conveniently called Seed), and evaluates the Grow expressions contained within that file.

Grow has a simple approach to dependency resolution : every value possesses a timestamp, and composing two or more values yields another value whose timestamp is the maximum of all its parameters. In Grow, everything is a value, from files and directories to dependency lists, functions, environments and programs.

Here is an example Seed file that compiles the ‘main.c’ file into the object file ‘main.o’, then links it to yield ‘main’.

tee $$arg:in {
  objects = hook cc [main.o] [main.c],
  exec = hook ld [main] [main.o]:in $objects,
  all = ($main:seq "All done"):in $exec
}

(x:f y is syntactic sugar for f x y, allowing any function to be treated as an infix operator).

In Grow, the recipes are not contained in the Seed file, and instead must be implemented as standalone executables accessible to Grow, so as to detect when a recipe changes and recompile the files that are built with that recipe. Essentially, Grow recipes are wrapper scripts that transform their arguments and call the true compiler. For example, here are the cc and ld wrappers, two 3-liners :

#!/bin/bash
obj="$1" ; shift ; src="$1"
gcc -c -o "$obj" "$src"
#!/bin/bash
bin="$1"; shift ; obj="$1"
gcc -o "$bin" "$obj"

Environments, lookups and virtual files

The $ function is used to retrieve values from this environment, so that $<expr> returns up the value of <expr> in the local environment, or the contents of the corresponding file if it doesn’t find it there. Here, <expr> can be any expression, including another variable lookup. In that sense, the $ function acts like the * operator in C, and can be stacked to provide multiple levels of dereferencing.

This explains the $$arg expression, which simply looks up the value of arg, then looks up that value again to yield the final result. For example, $$foo:in { foo = bar, bar = baz } will yield baz. The same is true of $$foo in a directory where the contents of the file ‘foo’ are ‘bar’ and the contents of ‘bar’ are ‘baz’, since files and variables are one and the same.

There are a few predefined functions and variables available to every Grow program :

Hooks and recipes

Another essential pervasive is the hook function. It is called thusly : hook <prog> <dsts> <srcs>, where prog is the name of an wrapper script in the current directory, and <srcs> and <dsts> are lists of variable names. The hook function returns an environment in which the destination names are associated to the new values as calculated by runnning the appropriate command. Of course, if all the destination files exist and are newer than both the recipe and ingredients, the command isn’t actually run and the destinations are directly read from disk.

There are no implicit rules for constructing files with Grow, as it would amount to defining an infinite amount of variable names. Instead, you may define your own functions to compile certain types of files and call those functions when needed.

Functions can be defined using the syntactic form (|<pattern> ~> <expression> |<pattern> ~> <expression> ...), which defines a function that tries to match its first argument to one of the patterns and returns the expression that corresponds to the first pattern matched (acting like an anonymous case expression).

Patterns can be lists, dictionaries or simple regular expressions to transform text. For example, fun main.c:in { fun = (|"(<prefix>.*).c" ~> $prefix) } will yield the value main. As you can see, a paren group that begins with <var> will bind the matched part to the variable var in the expression, allowing easy extraction of subfields).