Introducing “Train” — JavaScript-based build automation

Today i’m excited to talk about a new side project that Carlo and i have been working on in our spare time over the past couple of months: Train.

But let me start with some background.

Here at RemObjects Software, we use a sophisticated (but home grown) continuous integration system spread across several build servers to do automated builds and tests of our products. The build cycle for most of our products is intricate and lengthy, so of course it has to be automated: our “rofx-win” (more on what that one encompasses below) repository alone creates 16 different installers that we deploy, involving dozens of binaries that need to be built and countless additional tasks that need to be performed.

Until recently, we have mostly been using a commercial build automation tool that has served us well for the past almost 10 years, but we started running into its limitations — limitations not by any fault of the tool itself, but limitations imposed by the complexities of our builds.

Some of the problems included:

• It supported builds for Windows only, meaning for out Mac-based products, we had to use a different solution.*
• It provided a great IDE for creating build scripts, but that was also Windows only; i’m responsible for a lot of the build script maintenance here myself (because i’m the kind of anal type you need to be to keep these things clean and tidy ;), and i’m glad for any chance to avoid booting my Windows VM.
• But most importantly, it used a very verbose XML file format where every little adjustment to one part of the script caused thousands of small and inconsequential changes to the XML — changes that made merging a living hell (and we branch like crazy, here). It’s an eye-opening experience when after doing a cumbersome manual merge in Araxis, you get an error telling you that you overlooked a change in line 87,000. Yes, that’s line eight-seven-thousand. Not good.

So, Train

So we set out to create our build script engine to replace this tool, with the following goals in mind:

• We didn’t want to completely reinvent the wheel.
• It had to run on Windows and Mac OS X (and potentially elsewhere, such as Linux, though we don’t run it on that, ourselves).
• Build scripts had to be clean and easy to branch/merge.
• And, ideally, build scrips should be easily maintainable as plain text.

The result of these goals, lots of extensive spec’ing, and even more coding — mostly by Carlo — is Train, our new open source build script engine.

Train is based on JavaScript, and implemented using Oxygene and our already open-source RemObjects Script JavaScript engine. Simply put, Train is a tool that lets you take a JavaScript source file and run it. But Train is so much more: In addition to letting you use the full flexibility of JavaScript in our build script, Train adds:

• An extensive base API for common build tasks, from handling files operations over working with .ini or .xml files to building MSBuild, Delphi or Xcode projects. That API is of course expandable, both using .NET modules or directly in JavaScript.
• Sophisticated handling of variables, including access to environment variables and parameters passed to the build script, and in-string variable resolution.
• Flexible support for nesting and including sub-scripts.
• Great and readable logging, both to the console/stdout while a build is running, and to XML and rich HTML log files.

To give you an example, here are some parts of our “rofx-xcode” build script:

Among other things, you see this script calls run() on “DataAbstract.train”, which is a subscript that we actually ship with the product, so customers can run it standalone if they want to rebuild DA from source. The console output of “DataAbtract.train” while it is running might look something like this: As you see, it is nice and concise; you can see the flow of actions and JavaScrip functions the script is running thru, without any excess baggage (such as the extensive info Xcode dumps to the console).

Of course both XML and HTML log files can be generated as well, with the HTML being created from the XML using a standard XSL file included in Train. The output of a failed build might look like this:

 Of course you can tweak the XSL, or process the XML log file in your own way, if you like.

The Bigger Picture

With Train, we now have the last part of our continuous integration system under our own control, running our own internal software.

The rest (and much bigger and more complex part) of our build infrastructure, affectionately called “CI2”, is very interesting too, and i plan to write more about it in the future (and we are looking into possibly open-sourcing more parts of that, as well — but currently many things in CI2 are designed pretty specifically around our own needs).

Train fits in well with CI2, but is a completely independent piece of technology, and we hope that by making it available as a true open source project (under the BSD license), it will be helpful to many others looking for a clean and easy way to automate builds (or, really, any task).

Train in Action

At RemObjects, we have 5 main repositories that our build system handles: “rofx-win” (Data Abstract, RemObjects SDK and related tools, for all platforms but Xcode), “rofx-xcode” (you guessed it: Data Abstract and RemObjects SDK for Xcode), “oxygene”, “hydra” and “everwood”.

At the time of this writing, all but “rofx-win” have been converted to use Train — and “rofx-win” simply hasn’t because it’s a huge and complex build script that just takes time to convert/rewrite (and in the process get cleaned up and rethought). But between the other four repositories, Train has already been getting some good and thorough real life use, both on Windows and Mac OS X.

Get Train

There’s two ways to get Train, both from github.com/remobjects/train.

• If you’re feeling adventurous and/or want to get involved in extending and improving Train, clone a copy of our git repository (or fork your on copy — we’re looking forward to your merge requests!) and build it yourself. The clone will include RemObjects Script, so all you need is Oxygene for .NET (the free command line compiler will do).
• Alternatively, you can just grab the pre-build .zip distro with the latest compiled binaries, ready to run on Windows and Mac OS X (via Mono). We’re currently at revision 0.5, the same version running on our build systems.