Jenkins — Jenkins Configuration as Code (JCasC) together with JobDSL on Kubernetes

Introduction

There are many Jenkins installations outside. Most of them are deployed on servers and will be configured directly by Jenkins Admins or DevOps.
Backups will be done by saving the VM or the Jenkins XML configuration files.

Also, many instances will manage many jobs and nobody knows completely what is running. Or they are administrated by the developers directly, which results in configurations, that nobody can restore, especially if there is no backup available.

Problems with classic installations

All of those instances have different problems:

• Unflexible configurations, because everything must fit for everybody (all-in-one instances for whole companies)
• No real existing backup strategy is available (or absolutely untested)
• Or bad backup strategies like nightly VM backup, which results in loss of data during a day
• Updates can result in problems because huge instances often have a lot of plugins inside
• “Developer-managed” instances often have no backup or sometimes it crashes, but “nobody has done something”
• History of configurations and easy restore/revert is not possible in most cases, which includes also some kind of changelog/history with responsible users for changes are unavailable
• Switching the system to new servers or maybe new platforms is often a pain
• and many more…
• Problems with too many jobs and jobs management in general

Solution finding

Let’s make a wish-list for better Jenkins administration:

• Easy to deploy
• Easy to upgrade
• Easy to move from one platform to another
• Easy recovery if the system crashes
• Easy revert of changes
• Flexible configuration for projects
• Avoid too big installations and provide dedicated installations for dedicated teams
• Simple credentials management and update
• Traceable configuration
• Clear, versioned history of every setting
• Separate jobs management from Jenkins configuration
• Good backup strategy, which can recover everything anytime

This wish-list is very long, but how we can solve all of those things without installing hundreds of plugins, install a lot of servers, and so on?

Configuration

Jenkins Configuration as Code (JCasC)

First: What is JCasC?

It means, that all configuration is written as code. No backup of XML files, no “I did nothing” excuses, no “what have you done” accusations. No missing or too complex or unfeasible backup strategies.

You can install a plain, new Jenkins and let it configure from your code.

To solve a lot of problems, it is the best idea to use a GIT repository or something similar to manage your code. This helps to have a history, a backup and tracing about every change only by using a version control system (VCS).

The configuration itself will be written mostly as a YAML file. This makes it readable, you can diff versions and (if you want and if you use e.g. GIT) reviewable and approvable.

One very simple example:

In this example we configure the Jenkins system message, the URL of the Jenkins, which will be used, if we want to notify for example Github or Bitbucket about the build state, a global build library, sonar, and a simple job.

If we now start Jenkins with the JCasC plugin, we can tell where the file is located and Jenkins will load this file and configure itself. Every configuration, which is not part of the JCasC configuration (except the jobs) will be lost after a reload. But this is something, that we want to have because if we have misconfigured something, it is easy to reload this config and everything is like it was before. The JCasC plugin offers also a simple page, where we can click the button “Reload”.

To tell Jenkins, where this file is located, we can use the CASC_JENKINS_CONFIG environment variable.
You can find more information about the JCasC and the configuration at the Jenkins Configuration-as-Code Plugin site.

Jenkins JobDSL

Another help for better job management is the JobDSL plugin for Jenkins.
This is a simple DSL for defining jobs in Jenkins as code. It offers also some Groovy support.

To create a simple job (in our case a seed job to start all other jobs), you can define your jobs as closure.

To have a good starting point, you can add for example the following to you JCasC configuration on bottom:

Now the JCasC file contains the configuration and one initial job, which will be registered to Jenkins directly.
The JCasC part of this piece of code is the “jobs” and “-script:” declaration. The job() definition and its sub-elements are part of JobDSL.

So far we have a configuration and job management as code, which helps a lot for recreating our instances on other servers or clouds. By mixing those two technologies we have one file, which contains everything we need to start.

Jenkins-JobDSL-Remote tool

In the example above you see, that we have added a multiscm with two repositories to the new seed job. The first is a tool called jenkins-jobdsl-remote. By checkout this project to the target directory “jobdslscript”, we ensure, that we have always the newest version of this tool checked out. You also can clone it to your repository to have more control over the versions.
The second one is an example project, which contains more information about the jobs. For this example it contains also the project definition, in a real-world example, this should only contain the job-file.

The jenkins-jobdsl-remote script is a small Groovy script, that uses JobDSL to create jobs and parses simple JSON files with the job definition.
By defining the steps with the external execution of the jobdslscript/JenkinsJobDslRemote.groovy file (the first part of the path is the releativeTargetDirectory of the Git repository), we trigger this script. It looks into the workspace in the “jobdefinition” directory, if there is a “jenkins-dsl-jobs.json” file. If this file was found, it parses the file and manages the jobs.

With this mechanism, it is possible to define such a simple seed job for every project and to only define a repository, which contains the real jobs.
To make it much easier for developers to create jobs as code via JobDSL, this script helps with the JSON abstraction to create simple jobs in a repository without the need to learn Jenkins JobDSL.

A definition of the jobs, that the developers need can look like this:

This will create two jobs, one multibranch job, and one pipeline job.
The information, what to do in this job is defined in the Jenkinsfile.

Great so far. We have now reached the following goals:

• Our configuration is

1. traceable
2. reconfigurable
3. has a history with names who changed what via Git
4. separated from the jobs and in the hand of the DevOps

• Our jobs are

1. managed by a simple seed-job
2. developers can add their own jobs in a simple JSON file
3. everything is traceable via Git with a history
4. separated from the configuration and in the hand of the developers

• Our backup is our version control system and there is no need to have a separate backup of Jenkins because everything is rebuildable in zero time
• We are able to recreate new Jenkins instances with nearly the same configuration. We only have to create a new job repository for new teams, which must be configured in the JCasC file at the seed-job definition.

Deployment

Why Kubernetes?

Now, what about Kubernetes or why should we use it?

Kubernetes (in short K8S) is a platform, that was build to connect servers (nodes) to one big deployment cluster, that controls everything. It also was built to organize and orchestrate a lot of deployments, which can be a very hard way for classic installations, if you have more than 2 or 5 modules to install.

Kubernetes offers us also a very nice concept to separate projects or teams or installations with namespaces. Those namespaces are some kind of “isolated” environments. In the past, we had for those things different servers and may be different networks.

The result was a big zoo of infrastructure or VMs, which became unmanageable over time.

This doesn’t mean, that Kubernetes is easy to handle. To have a production-ready solution with Kubernetes you have to manage also a lot of things. But all of those things are written as simple YAML files, which can also be used as configuration as code for other systems.

It is always a good idea to automate also the base platform as much as possible.

To manage the cluster (or many clusters) it can be also helpful to look into tools like:

• Kublr
• Rancher
• and others

(see here for a great list of Kubernetes related tools: https://github.com/ramitsurana/awesome-kubernetes#cluster-manager)

There is also a small “version” of Kubernetes called “K3S” (or “K3D” for better management), which is developed by Rancher.
K3S installed a complete Kubernetes as one binary file which fewer hardware requirements. This makes it also possible to use it on a local machine for testing.

So, this all sounds interesting, but how to deploy Jenkins on Kubernetes?

Kubernetes uses a Docker container to deploy applications. This can be done by applying some deployment definitions, written in YAML, to Kubernetes.

But this is often not such intuitive as you hope.

To solve this, there is Helm with its “Helm Charts”. Those “Helm Charts” are templated deployment descriptions, which can be used to deploy applications on Kubernetes. This means, that a generic deployment description was written in a template file and depending on the configuration, which will be provided as “values.yaml” file the application can be deployed with the personal needs (in most cases).

Thankfully Jenkins also offers such a “Helm Chart”.

So theoretically we can install helm, kubectl, and then installing Jenkins.
But we want to combine JCasC, JobDSL, and external job management. Also, we maybe want to have deployments of our applications in separate namespaces, controlled by Jenkins and we want to have routing-rules per namespace which should not collide with others from other namespaces.

K8S-JcasC-Management tool

With the k8s-jcasc-management-go tool, it is possible to combine all of those things with predefined templates, an IP management to avoid collisions, and with a simple shell script, which does everything from creating a project template, installing/uninstalling on Kubernetes and managing the secrets.

Here a simple picture of the flow from the tools site:

The k8s-jcasc-management-go tool offers a configuration so define some recurring things and definitions of the folders, where your project configuration and secrets should be stored until some basic things like container definition and so on.

It is recommended to create a file “k8s_jcasc_custom.cnf” under the config folder, which defines the path to an own configuration file. With that, you can check out the tool, upgrade the tool, and store your own configuration in a place, where you have the control over.

This file contains something like this and is excluded via .gitignore for check-ins:

What does this mean?
It means, that you can have your own copy of the config file in the specified path under the “K8S_MGMT_ALTERNATIVE_CONFIG_FILE” variable and that you can use it only as an overlay to change only relevant configurations. The system primarily loads its own configuration in this case and then overwrites all values from your config file.
This is helpful if new configurations are available or to know, what is needed by the own system instead of the default.

It also contains a copy of the Jenkins Helm Charts, which are very slightly manipulated, to add some features.

After this setup was done, the tool is very easy to use:
simply call the “go run k8s-jcasc-mgmt.go” file from your console (Linux, WSL under Windows, Powershell or CMD, Mac console…).

When the tool was started, it gives you the possibility to select the action that you want to do.

One of the first steps should be to create a new project. It asks for some parameters like IP address, namespace, target directory of the project (should be the same as the namespace to keep it simple), and so on. Then it creates the configuration of this project as Jenkins configuration as code files.
This tool works only with remote configuration, which means, that you have to check-in your newly created project configuration first into a VCS. This repository needs also public read-access that Jenkins is able to read the configuration directly from this URL.

The next step should be to configure your secrets. To do this, start “k8s-jcasc-mgmt” and select “decryptSecrets” (initial password is “admin”). Now it decrypts the secrets file into a readable “secrets.sh” file, which contains some examples of how to configure the secrets.
After you have changed your things, you can restart the script and select “encryptSecrets”, which asks you for your new password and stores the new encrypted secrets file. It also deletes the plain “secrets.sh” file, that you can not check-in this file, which can be a security risk.

Before we start to install something, it is necessary, that the following tools are installed:

• helm
• kubectl

Please install them, if you haven’t already.

Now to install Jenkins on a cluster, simply call “k8s-jcasc-mgmt”, select install, type your directory, and namespace into the console and Jenkins should start.

Updates

• Follow-up article: https://ragin.medium.com/kubernetes-and-ci-cd-how-to-integrate-in-your-development-process-9b483b194975
• 26.06.2020 — Updated article from bash version to go version, because the bash version is now deprecated.