At Okta, we’ve gone through many iterations of using Jenkins to build and test our software. We use a number of tools to make sure our code works properly, and we like to have Jenkins manage these. The list would be familiar to anyone using the Java environment; PMD, Cobertura, unit and functional tests with JUnit, Selenium tests with testNG, and also some more exotic tools like BURP Security scanner, MogoTest, and SLAMD.

We quickly found that manually installing and configuring Jenkins when we needed a new server for a given task or a special project, or rebuilding an existing server which had crashed or been eaten by Amazon was incredibly time-consuming and error-prone. We implemented a simple way of managing the Jenkins server configuration and job configurations for a given instance with code. I’m going to walk through the approach we took (simplified a bit) as an example of how to do this.

We happen to use Python and Jinja2, however this approach could be taken with any template language that has simple variable replacement and logical controls for looping and conditionals.

So let’s get started.

Library and Package Prerequisites

This assumes that we are running on CentOS, however, we can and do run these same things on Windows and Mac.

Python 2.6

We standardize on this version, but others will work, too. For tutorial reasons, though, I'll reference this specific libraries we use. If you don't have Python installed, or you have a different version, you can get 2.6 here: http://python.org/ftp/python/2.6.6/Python-2.6.6.tgz

 

 wget http://python.org/ftp/python/2.6.6/Python-2.6.6.tgz tar fxz Python-2.6.6.tgz cd Python-2.6.6 ./configure make sudo make altinstall sudo ln -s /usr/local/bin/python2.6 /usr/bin/python2.6  

 

setuptools

setuptools is needed for Jinja2 installation.

 

 wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg sudo sh setuptools-0.6c11-py2.6.egg

 

Jinja2

This package can be retrieved from here: http://pypi.python.org/packages/source/J/Jinja2/Jinja2-2.5.5.tar.gz

 

 wget http://pypi.python.org/packages/source/J/Jinja2/Jinja2-2.5.5.tar.gz tar fxz Jinja2-2.5.5.tar.gz cd Jinja2-2.5.5 sudo python2.6 setup.py install

 

Jenkins: The Heart of It

In order to dynamically create a Jenkins instance, it is necessary to understand which files are where and how they are used.

Jenkins is a great piece of software, and part of it's greatness is the ease of use via the Jenkins web gui, especially for people just starting out. But at it's heart, Jenkins uses configuration files defined with xml to control the application. Editing these xml configuration files has the same effect as editing Jenkins through the gui. It's easier to generate xml files with code, but usually less complex than using the built-in Jenkins APIs. So this is the approach we took for managing our Jenkins instances.

When Jenkins is fired up with java -jar jenkins.war or some more involved variant of that, it creates a directory called .jenkins which is referred to with the environment variable called JENKINS_HOME. We will be working with the files created in this directory to control how the server and corresponding jobs are defined.

Go ahead and download the latest stable Jenkins war from the open source project here: http://jenkins-ci.org/

We'll walk through the contents of the directory created by Jenkins. This is key to understanding how to create a specific Jenkins instance and associated jobs with code.

Once downloaded, set the JENKINS_HOME variable to something called .jenkins in your current working directory:

 

 export JENKINS_HOME=`pwd`/.jenkins

 

By default, it will be created in a file called .jenkins in the users home directory, but we don't want that, we want it to be in the current working directory in a hidden directory called .jenkins, alongside the jenkins.war.

Start up the Jenkins web application:

 

 java -jar jenkins.war

 

Let's walk through the structure.

You should see something like this:

 

 $ cd .jenkins [.jenkins]$ ls -al total 32 drwxr-xr-x 10 dlumma staff 340 Mar 19 15:22 . drwxr-xr-x 4 dlumma staff 136 Mar 19 15:22 .. -rw-r--r-- 1 dlumma staff 159 Mar 19 15:22 hudson.model.UpdateCenter.xml -rw------- 1 dlumma staff 1675 Mar 19 15:22 identity.key drwxr-xr-x 2 dlumma staff 68 Mar 19 15:22 jobs -rw-r--r-- 1 dlumma staff 907 Mar 19 15:22 nodeMonitors.xml drwxr-xr-x 16 dlumma staff 544 Mar 19 15:22 plugins -rw-r--r-- 1 dlumma staff 64 Mar 19 15:22 secret.key drwxr-xr-x 3 dlumma staff 102 Mar 19 15:22 userContent drwxr-xr-x 30 dlumma staff 1020 Mar 19 15:22 war

 

This is the skeleton directory structure of what Jenkins uses to drive itself. More files and directories are added as more functionality and configurations are specified. The directories we deal with and find most useful are 1) the jobs directory (this defines all the jobs shockingly) and the 2) plugins directories (where all the plugins are defined, also shocking, I know). By default, there is nothing defined in the jobs directory, but there are already a few plugins installed like ant maven, ssh and subversion.

Aside from things in the jobs and plugins directories, the other file we manipulate a lot is called config.xml. This file resides in the top level under .jenkins. This is the main configuration file for the Jenkins server. We can get Jenkins to generate this file by modifying the http://jenkinsinstance/configure page through the ui. Navigate to the configure page, and add a System Message "My Awesome Jenkins Server", and click Save. The config.xml should now exist in the .jenkins directory.

 

$emacs .jenkins/config.xml shows:

 

<?xml version='1.0' encoding='UTF-8'?> <hudson> <disabledAdministrativeMonitors/> <version>1.456</version> <numExecutors>2</numExecutors> <mode>NORMAL</mode> <useSecurity>true</useSecurity> <authorizationStrategy/> <securityRealm/> <projectNamingStrategy/> <workspaceDir>${ITEM_ROOTDIR}/workspace</workspaceDir> <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir> <systemMessage>My Awesome Jenkins Server</systemMessage> <jdks/> <viewsTabBar/> <myViewsTabBar/> <clouds/> <slaves/> <quietPeriod>5</quietPeriod> <scmCheckoutRetryCount>0</scmCheckoutRetryCount> <views> <hudson.model.AllView> <owner reference="../../.."/> <name>All</name> <filterExecutors>false</filterExecutors> <filterQueue>false</filterQueue> <properties/> </hudson.model.AllView> </views> <primaryView>All</primaryView> <slaveAgentPort>0</slaveAgentPort> <label></label> <nodeProperties/> <globalNodeProperties/> </hudson>  

After understanding the basic structure of the files which Jenkins uses to determine it's configuration and functionality it is possible to go about defining it with source code.

Jenkinizer

Now feel free to blow away your Jenkins instance and all files associated with it and clone the repository setup for this blog post by Okta here:

 

git clone [email protected]:/okta/jenkinizer

Our jenkinizer repo has four kinds of files: python configuration files, xml template files, static resource files, and scripts. The structure looks like this (it is a simplified version of our actual internal code and only contains a sub-set of code related to selenium testing for clarity):

 

jenkinizer/
 -- config/
 ---- servers/
 ------ selenium/
 -------- selenium_server.py
 ---- templates/
 ------ selenium/
 -------- selenium_server_config.xml
 -------- selenium_setup.xml
 -------- selenium_suite.xml
 -- install/
 ---- .jenkins/
 ------ plugins/
 -------- (a bunch of plugins we have installed)
 ------ hudson.plugins.emailext.ExtendedEmailPublisher.xml
 ---- jenkins-env.sh
 ---- jenkins.sh
 ---- jenkins.war
 -- mkconfig.py

Directories

config

This directory contains all the template and job configurations.

install

Static resources and initialization scripts for starting Jenkins are under install.

A master script called mkconfig.py resides in the main directory. Let's walk through what this does.

 class ServerOptionParser(OptionParser):     def __init__(self):         OptionParser.__init__(self)         self.add_option("-t", "--jenkins-type", action="store", dest="jenkins_type", default="selenium",                          help="Type of server to generate." )         self.add_option("-j", "--jenkins-jobset", dest="jenkins_jobset", default="integration",                           help="Specific host this Jenkins will be deployed to.")         self.add_option("-o", "--install-dir", dest="install_dir", default="/ebs/ci-build/tools/jenkins",                           help="Location to install the newly generated server.")         self.add_option("-b", "--jenkins-branch", dest="jenkins_branch", default="release",                           help="Branch to use in testing.")         self.add_option("-e", "--recipient-email", dest="recipient_email", default=" ",                           help="Email recipient")         self.add_option("-d", "--discard-old-builds", action="store_true", dest="discard_old_builds", default=False,                           help="If set, old builds will be discarded.")

This is a simple command line tool made from the lovely OptionParser library. We define a few parameters and arguments. The most important ones are the jenkins-type and jenkins-jobset. In this example I am showing a distilled version of our selenium servers. So the jenkins-type will be selenium and the jenkins-jobset will be either integration, pre-integration, or serial. We have a lot of other kinds of jenkins servers which I'm not including here for things like building and testing mobile, windows artifacts, unit and functional builds, master/ slave configurations and dashboards. If you want to make servers of different kinds, you could follow the pattern here by creating a new jenkins type and giving it a group of jobsets.

The next few methods are used for keeping test results when he upgrade the Jenkins instance itself. We sometimes don't care, but sometimes do want to preserve the test results collected so far, and add a new job, or alter a setting slightly. These methods handle that.

 def make_selected_archive(backup_file_name, base_dir, files, gzip = True):     backup_file = tarfile.open(backup_file_name, 'w:gz' if gzip else "w")     current_dir = os.getcwd()     os.chdir(base_dir)     for artifact in files:         if os.path.exists(artifact):             backup_file.add(artifact)     backup_file.close()     os.chdir(current_dir)  def restore_archive(backup_file_name, base_dir, gzip = True):     backup_file = tarfile.open(backup_file_name, "r:gz" if gzip else "r")     backup_file.extractall(base_dir)     backup_file.close()  def backup_old_builds(jobs_dir, job_names):     archives = {}     for job in job_names:         job_dir = jobs_dir + "/" + job         if os.path.exists(job_dir):             print "Backing up old builds in %s" % (job_dir)             backup_file_name = tempfile.gettempdir() + "/" + job + ".tar.gz"             make_selected_archive(backup_file_name, job_dir, ["./builds", "./lastStableBuild", "./lastSuccessfulBuild", "./nextBuildNumber"])             archives[job] = backup_file_name     return archives

Then comes the main functionality, which generated the configuration files for this Jenkins server and its jobs.
We get the type, jobset, installation_dir and email to be used in this instance.

 def build_server(options, base_path = os.getcwd()):      # We categorize Jenkins instances depending on the jobs they run.  We have Jenkins instances that run selenium,     # run unit and functional tests, build artifacts and deploy artifacts. This defined the kind of Jenkins we are     # deploying.     jenkins_type = options.jenkins_type      # The set of jobs to deploy.     jenkins_jobset = options.jenkins_jobset      # The location to install this Jenkins instance to on the filesystem.     install_dir = options.install_dir      # Email.     recipient_email = options.recipient_email

We load up the template files defined for our type of server.

     # Make our python job and server configuration files reachable.     sys.path.extend([base_path + "/config/servers"])

We import the specific dict defining the specified jobset and jenkins type.

     # Define the template environment.     template_env = Environment(loader=FileSystemLoader(base_path + '/config/templates/' + jenkins_type + '/'))

We define the installation locations.

     # Import the specific server and job data for this jenkins type     # and this jobset.     server_config = __import__(jenkins_type).__dict__[jenkins_jobset]     server_config["env"]["EMAIL"] = recipient_email     server_config["env"]["BRANCH"] = options.jenkins_branch

If specified, we backup existing archives and then delete the current Jenkins installation.

     jenkins_dir = install_dir + "/.jenkins"     jobs_dir = jenkins_dir + "/jobs"      # Backup archived files and     # remove the old instance if it exists.     archives = {}     if os.path.exists(install_dir):         if not options.discard_old_builds:             archives = backup_old_builds(jobs_dir, [job["job_name"] for job in server_config["jobs"]])         shutil.rmtree(install_dir)

We copy all of the static resource files to the installation directory and print out an informational message.

     # Copy all of the static Jenkins files to the installation directory     shutil.copytree(base_path + "/install", install_dir)      print "Generating configuration for jenkins type %s (writing to %s)" % (jenkins_type, install_dir)

We generate the main Jenkins configuration file with the given configuration file and server configuration.

     # Create the jenkins main config file.     with open(jenkins_dir + "/config.xml", "w") as f:         t = template_env.get_template("server-config.xml")         f.write(t.render(server_config))

And finally, we generate the jobs configuration files, write them to the installation directory and restore any archived files.

     # Create the jobs config files.     for job in server_config["jobs"]:         job_name = job[ "job_name"]         job_template = job["template_file" ]         job.update(server_config)          print "Creating job %s from %s" % (job_name, job_template)         job_dir = jobs_dir + "/" + job_name         os.makedirs(job_dir)         with open(job_dir + "/config.xml", "w") as f:             t = template_env.get_template(job_template)             f.write(t.render(job))          if archives.has_key(job_name):             print "Restoring old builds for %s" % (job_name)             restore_archive(archives[job_name], job_dir)

Now let's look at the files used. The template files under config/templates/selenium are xml files with simple variables defined in the selenium.py configuration file. These were originally copied from the Jenkins installation and edited with variables for dynamic rendering based on the data defined in the Python files.

The main job data is defined in config/servers/selenium.py. Each jobset is defined with a dict built from get_default() which contain the keys env, os, chained, views and jobs. The jobs key references another list which contains dicts which specific job data such as job_name, template_file, max_builds_to_keep, repo, and others.

Updating an existing Jenkins selenium server with the integration jobset would look like this:

 

$ python2.6 mkconfig.py -t selenium -j integration Generating configuration for jenkins type selenium (writing to /ebs/ci-build/tools/jenkins) Creating job okta.build from selenium-setup.xml Creating job core.chrome from selenium-suite.xml Creating job apps.chrome from selenium-suite.xml Creating job plugin.apps.chrome from selenium-suite.xml Creating job core.firefox.latest from selenium-suite.xml Creating job apps.firefox.latest from selenium-suite.xml Creating job plugin.apps.firefox.latest from selenium-suite.xml Creating job core.ie.8 from selenium-suite.xml Creating job apps.ie.8 from selenium-suite.xml Creating job plugin.apps.ie.8 from selenium-suite.xml Creating job core.firefox.3.6 from selenium-suite.xml Creating job apps.firefox.3.6 from selenium-suite.xml Creating job plugin.apps.firefox.3.6 from selenium-suite.xml Creating job core.ie.9 from selenium-suite.xml Creating job apps.ie.9 from selenium-suite.xml Creating job plugin.apps.ie.9 from selenium-suite.xml

Now go to the install directory and start it up.

 

$ cd /ebs/ci-build/tools/jenkins
 $ ./jenkins.sh
 JENKINS_HOME=/ebs/ci-build/tools/jenkins/.jenkins

Tail the log to make sure everything is running smoothly.

 

$ tail -f jenkins.log Running from: /ebs/ci-build/tools/jenkins/jenkins.war webroot: EnvVars.masterEnvVars.get("JENKINS_HOME") Apr 1, 2012 10:04:26 PM winstone.Logger logInternal INFO: Beginning extraction from war file Jenkins home directory: /ebs/ci-build/tools/jenkins/.jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME") Apr 1, 2012 10:04:29 PM winstone.Logger logInternal INFO: HTTP Listener started: port=8080 Apr 1, 2012 10:04:29 PM winstone.Logger logInternal INFO: AJP13 Listener started: port=8009 Apr 1, 2012 10:04:29 PM winstone.Logger logInternal INFO: Winstone Servlet Engine v0.9.10 running: controlPort=disabled Apr 1, 2012 10:04:30 PM jenkins.InitReactorRunner$1 onAttained INFO: Started initialization Apr 1, 2012 10:04:32 PM jenkins.InitReactorRunner$1 onAttained INFO: Listed all plugins Apr 1, 2012 10:04:32 PM jenkins.InitReactorRunner$1 onAttained INFO: Prepared all plugins Apr 1, 2012 10:04:32 PM jenkins.InitReactorRunner$1 onAttained INFO: Started all plugins Apr 1, 2012 10:04:32 PM jenkins.InitReactorRunner$1 onAttained INFO: Augmented all extensions Apr 1, 2012 10:04:36 PM org.apache.sshd.common.util.SecurityUtils$BouncyCastleRegistration run INFO: Trying to register BouncyCastle as a JCE provider Apr 1, 2012 10:04:36 PM org.apache.sshd.common.util.SecurityUtils$BouncyCastleRegistration run INFO: Registration succeeded Apr 1, 2012 10:04:36 PM jenkins.InitReactorRunner$1 onAttained INFO: Loaded all jobs Apr 1, 2012 10:04:36 PM org.jenkinsci.main.modules.sshd.SSHD start INFO: Started SSHD at port 52810 Apr 1, 2012 10:04:36 PM jenkins.InitReactorRunner$1 onAttained INFO: Completed initialization Apr 1, 2012 10:04:37 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@2be3d80c: display name [Root WebApplicationContext]; startup date [Sun Apr 01 22:04:37 PDT 2012]; root of context hierarchy Apr 1, 2012 10:04:37 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@2be3d80c]: org.springframework.beans.factory.support.DefaultListableBeanFactory@1e94b0ca Apr 1, 2012 10:04:37 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1e94b0ca: defining beans [authenticationManager]; root of factory hierarchy Apr 1, 2012 10:04:37 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@23a65a18: display name [Root WebApplicationContext]; startup date [Sun Apr 01 22:04:37 PDT 2012]; root of context hierarchy Apr 1, 2012 10:04:37 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@23a65a18]: org.springframework.beans.factory.support.DefaultListableBeanFactory@29cc3436 Apr 1, 2012 10:04:37 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@29cc3436: defining beans [filter,legacy]; root of factory hierarchy Apr 1, 2012 10:04:37 PM hudson.TcpSlaveAgentListener INFO: JNLP slave agent listener started on TCP port 7070 Apr 1, 2012 10:04:42 PM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in okta.build. Triggering #1 Apr 1, 2012 10:04:48 PM hudson.WebAppMain$2 run INFO: Jenkins is fully up and runnin Apr 1, 2012 10:04:58 PM hudson.model.DownloadService$Downloadable doPostBack INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller Apr 1, 2012 10:04:58 PM hudson.model.DownloadService$Downloadable doPostBack INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller Apr 1, 2012 10:04:58 PM hudson.model.DownloadService$Downloadable doPostBack INFO: Obtained the updated data file for hudson.tools.JDKInstaller Apr 1, 2012 10:05:01 PM hudson.model.UpdateSite doPostBack INFO: Obtained the latest update center data file for UpdateSource default Apr 1, 2012 10:05:42 PM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in okta.build. Triggering #2 Apr 1, 2012 10:06:42 PM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in okta.build. Triggering #3

Open your browser and see the newly created Jenkins instance at: http://localhost:8080

This is a simple and effective way to manage multiple Jenkins instances intended to do different kinds of things. It is more maintainable than configuring a new instance through the gui, which is very error prone and takes a lot of time. Yet it is straightforward and doesn't involve any advanced programming. For managing a lot of Jenkins instances doing a lot of different things, this approach has worked well for us.