Skip to content
Back to projects
BaxterDec 2023

Jenkins Platform on Kubernetes

Made Jenkins on Kubernetes safe to upgrade and recover with Configuration as Code, multi-level backups, Kaniko builds, and parametrized Bitbucket webhooks.

DevOpsKubernetes
Bitbucket push triggers a parametrized Jenkins job on Kubernetes, with config as code and backups

Context: Baxter International (Healthcare)

Problem

Jenkins was the team's main CI tool, but running it was fragile. Upgrades were frequent, slow, and often brought breaking changes, and each one risked losing important data. Jenkins wasn't well integrated with Kubernetes. When a Jenkins server was accidentally deleted, the company lost a key tool, which made a robust backup and recovery approach a priority.

Builds were also hard to control. The native Bitbucket plugin triggered a build on every push to any branch, couldn't pass the branch or repository to the job, and pushed the team toward one job per repository.

Approach

  • Jenkins on Kubernetes. I deployed Jenkins with a controller and agents on Kubernetes and managed authentication, roles, and jobs.
  • Configuration as Code. I introduced the Jenkins Configuration as Code (JCasC) plugin and built an upgrade framework around it, so upgrades and compatibility fixes became repeatable instead of manual.
  • Multi-level backups. A Kubernetes CronJob backs up the key Jenkins files on a schedule. I wrote the disaster recovery process and trained the rest of the team to restore from the backups.
  • Kaniko builds. Container images are built with Kaniko, which avoids privileged Docker-in-Docker on the cluster. Pipelines are written in Groovy Jenkinsfiles.
  • Parametrized Bitbucket webhooks. I replaced the native Bitbucket plugin with the Generic Webhook Trigger plugin. JSONPath expressions extract the branch and repository name from the webhook payload and pass them to the job as parameters. Builds run only for the branches that matter, and one job serves multiple repositories in the same project.

Bitbucket webhook payload is parsed into parameters for a single Jenkins job running on Kubernetes

Impact

  • Upgrades went from risky manual work to a repeatable, versioned process.
  • Jenkins can be restored from backup, and the whole team knows how to do it.
  • Builds trigger only on relevant branches, from one parametrized job instead of one job per repository.

Write-up

Stack

Jenkins · JCasC · Kaniko · Groovy · Bitbucket · Kubernetes