Managing Results from Continuous Testing

One of the most important requirements to ensure before deploying to production is that to-be shipped artifacts have adequately passed a suite of tests that define correct behavior. If code in a particular git commit has vaulted this bar, we consider that commit “blessed” and a candidate to ship to production. Each week, we choose one candidate commit — likely the most recent — to push onto the live site.

To easily and visually make a determination about what commits are “blessed,” we created a tool called Beatify* which aggregates data from the different continuous integration tests we run. In each row of the display, Beatify lists each commit merged to the mainline branch as well as its performance on all of our tests. Subsequent tests run conditional on the previous tests’ success so if, say, the functional test suite fails the selenium tests will not run on that commit, and the entire test process will start anew on the most recent commit to mainline.

To be considered a candidate, a commit must pass the baseline unit and functional tests as well as a sufficiently high number of selenium tests. We run several different suites of selenium tests (with on the order of one hundred tests per suite) across a handful of different browsers.

Paging in the interface groups commits by those that were pushed to production in a certain release. So, for instance, paging back once would show all commits and test results shipped in the previous release; twice would give the commits from the second most recent release; etc. Beatify knows how to group releases by reading tags that are dropped on certain commits, signifying that a commit was pushed to production.

There are a few other interfaces besides the main Beatify view. These other views contain diagnostic tools relating to our Jenkins clusters and other Beatify-like UIs that show different subsets of the test/commit database.

* The name Beatify derives from the process of Beatification in the Catholic Church, which is a blessing bestowed on an individual that is prerequisite for sainthood.