Jenkins basics

Introduction

Jenkins is an open-source software system written in Java , designed to support the continuous integration process of software.

It was forked in 2008 from the Hudson project owned by Oracle, by its main author, Kohsuke Kawaguchi.

It is distributed under the MIT license.

It automates part of the software development process that does not require human involvement, providing continuous integration functions.

It runs in a servlet container, for example Apache Tomcat.

It supports version control system tools, including AccuRev, CVS, Subversion , Git , Mercurial, Perforce, Clearcase and RTC.

It can build projects using Apache Ant and Apache Maven , as well as run arbitrary shell scripts and batch files Windows .

A build can be started in various ways, for example by a commit event in the version control system, on a schedule, by a request to a specific URL, or after another build in the queue finishes.

Jenkins capabilities can be extended with plugins.

Access control is implemented in two ways: user authentication and authorization.

Protection against external threats is supported, including CSRF attacks and malicious builds.

In 2011 Jenkins received an InfoWorld award as the best open-source project.

By default it listens on TCP port 8080

Jenkins Docker Agent

A relatively new way to create agents is to run them in Docker containers. To do this, you need to add the Docker plugin in Jenkins

Plugin Manager → Cloud Providers → Docker

Configuring Jenkins to run a docker agent, image from www.aredel.com
After installing the Docker plugin
www.aredel.com

Configuration is done through

Manage Jenkins → Manage nodes and clouds → Configure Clouds

Configuring Jenkins to run a docker agent, image from www.aredel.com
Configure Clouds
www.aredel.com

Let's set up a Docker server on a separate Ubuntu host on the local network and then return to configuring Jenkins

IP address of my Ubuntu host 10.15.253.223

First of all, you need to install Docker

Then on the Docker server you need to allow connections to the Docker API.

To do this, you need to open port 2375

You also need to open ports from 32768 to 60999, which will be used for agents ( instructions for Ubuntu )

sudo vi /lib/systemd/system/docker.service

# ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock

sudo vi /lib/systemd/system/docker.service
sudo service docker status -l

● docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2023-01-09 12:00:07 EET; 3min 47s ago TriggeredBy: ● docker.socket Docs: https://docs.docker.com Main PID: 7709 (dockerd) Tasks: 10 Memory: 29.9M CPU: 862ms CGroup: /system.slice/docker.service └─7709 /usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock Nov 17 09:24:04 andrei-virtual-machine dockerd[7709]: time="2023-01-09T12:00:05.488100986+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc Nov 17 09:24:04 andrei-virtual-machine dockerd[7709]: time="2023-01-09T12:00:05.850772393+02:00" level=info msg="[graphdriver] using prior storage driver: overlay2" Nov 17 09:24:04 andrei-virtual-machine dockerd[7709]: time="2023-01-09T12:00:05.934673941+02:00" level=info msg="Loading containers: start." Nov 17 09:24:04 andrei-virtual-machine dockerd[7709]: time="2023-01-09T12:00:06.450509708+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" Nov 17 09:24:04 andrei-virtual-machine dockerd[7709]: time="2023-01-09T12:00:06.689935636+02:00" level=info msg="Loading containers: done." Nov 17 09:24:04 andrei-virtual-machine dockerd[7709]: time="2023-01-09T12:00:06.990191356+02:00" level=info msg="Docker daemon" commit=20.10.12-0ubuntu4 graphdriver(s)=overlay2 version=20.10.12 Nov 17 09:24:04 andrei-virtual-machine dockerd[7709]: time="2023-01-09T12:00:06.990408478+02:00" level=info msg="Daemon has completed initialization" Nov 17 09:24:04 andrei-virtual-machine systemd[1]: Started Docker Application Container Engine. Nov 17 09:24:04 andrei-virtual-machine dockerd[7709]: time="2023-01-09T12:00:07.109669043+02:00" level=info msg="API listen on [::]:2375" Nov 17 09:24:04 andrei-virtual-machine dockerd[7709]: time="2023-01-09T12:00:07.122519364+02:00" level=info msg="API listen on /var/run/docker.sock"

Let's check whether the Docker API is available using curl

curl http://localhost:2375/version

{"Platform":{"Name":""},"Components":[{"Name":"Engine","Version":"20.10.12","Details":{"ApiVersion":"1.41","Arch":"amd64","BuildTime":"2022-03-07T15:57:50.000000000+00:00","Experimental":"false","GitCommit":"20.10.12-0ubuntu4","GoVersion":"go1.17.3","KernelVersion":"5.15.0-57-generic","MinAPIVersion":"1.12","Os":"linux"}},{"Name":"containerd","Version":"1.5.9-0ubuntu3.1","Details":{"GitCommit":""}},{"Name":"runc","Version":"1.1.0-0ubuntu1.1","Details":{"GitCommit":""}},{"Name":"docker-init","Version":"0.19.0","Details":{"GitCommit":""}}],"Version":"20.10.12","ApiVersion":"1.41","MinAPIVersion":"1.12","GitCommit":"20.10.12-0ubuntu4","GoVersion":"go1.17.3","Os":"linux","Arch":"amd64","KernelVersion":"5.15.0-57-generic","BuildTime":"2022-03-07T15:57:50.000000000+00:00"}

Let's check availability from the network

curl http://10.15.253.223:2375/version

{"Platform":{"Name":""},"Components":[{"Name":"Engine","Version":"20.10.12","Details":{"ApiVersion":"1.41","Arch":"amd64","BuildTime":"2022-03-07T15:57:50.000000000+00:00","Experimental":"false","GitCommit":"20.10.12-0ubuntu4","GoVersion":"go1.17.3","KernelVersion":"5.15.0-57-generic","MinAPIVersion":"1.12","Os":"linux"}},{"Name":"containerd","Version":"1.5.9-0ubuntu3.1","Details":{"GitCommit":""}},{"Name":"runc","Version":"1.1.0-0ubuntu1.1","Details":{"GitCommit":""}},{"Name":"docker-init","Version":"0.19.0","Details":{"GitCommit":""}}],"Version":"20.10.12","ApiVersion":"1.41","MinAPIVersion":"1.12","GitCommit":"20.10.12-0ubuntu4","GoVersion":"go1.17.3","Os":"linux","Arch":"amd64","KernelVersion":"5.15.0-57-generic","BuildTime":"2022-03-07T15:57:50.000000000+00:00"}

Let's return to Jenkins and configure the cloud

Configuring Jenkins to run a docker agent, image from www.aredel.com
Configure Clouds
www.aredel.com

Now let's configure the agent, keeping in mind

Requirements:

The docker container's mapped SSH port, typically a port on the docker host, has to be accessible over network from the master. Docker image must have sshd installed. Docker image must have Java installed. Log in details configured as per ssh-slaves plugin.

You can make up any name, for example demo-docker-slave

I use my own Docker image. You can study the code here .

You can either use the link to my image andreissh/jenkins-docker-slave:latest or build your own image and push it to your Docker Hub following the instructions

Configuring Jenkins to run a docker agent, image from www.aredel.com
Configuring the agent
www.aredel.com

Do not forget to specify Remote File System Root. In this example it is

/home/jenkins

I also recommend setting the Instance Capacity parameter. For example, you can limit it to two. If you do not set this parameter, Jenkins will launch as many agents as it can. If something goes wrong, you will find a list of a large number of inactive agents.

As the Connect Method you can choose SSH and connect with a password.

Configuring Jenkins to run a docker agent, image from www.aredel.com
Configuring the agent
www.aredel.com
Configuring Jenkins to run a docker agent, image from www.aredel.com
Configuring the agent
www.aredel.com
Configuring Jenkins to run a docker agent, image from www.aredel.com
Configuring the agent
www.aredel.com

Now you need to create a job or update an existing one so that it runs on the agent

Running a Jenkins job on a docker agent, image from www.aredel.com
Restrict → Label Expression
www.aredel.com

The job itself is elementary: print text with the echo

After successful completion, the logs show that the job ran remotely on the agent demo-docker-slave-xxxxxxxxxxxxx the agent id is generated at creation.

Started by user andrei Running as SYSTEM Building remotely on demo-docker-slave-000gd8ottq9e0 on docker (demo-docker-slave) in workspace /home/jenkins/workspace/job_for_docker_agent [job_for_docker_agent] $ /bin/sh -xe /tmp/jenkins17605261416739110587.sh + echo Job for docker agent is done! Job for docker agent is done! Finished: SUCCESS

Let's add another template for the docker agent. Now the connection will not be over SSH but via JNLP

Manage Jenkins → Manage nodes and clouds → Configure Clouds → Docker Agent Template → Add Docker Template

We will use the image jenkins/jnlp-slave

The old jenkinsci/jnlp-slave image is outdated and not recommended for use

Requirements:
Jenkins master has to be accessible over network from the container.
Docker image must have Java installed.
Docker image must launch agent.jar by itself or using the EntryPoint Arguments below.

On the Jenkins master

Configure Global Security → Agents → TCP port for inbound agents

You need to set Fixed 50000

Port 50000 was opened in advance in the container with the ports parameter in the docker-compose.yml file

Automatic job triggering

Let's look at a job that uses a repository on GitHub

Configure → Source Code Management → Build Triggers

First, let's select Poll SCM

The frequency of repository update checks is set in the style of cron

H/5 * * * *

Then you need to commit to the repository connected to this job and wait about five minutes.

Started by an SCM change Running as SYSTEM Building remotely on docker_slave_ssh-000i0w2937lrg on docker_ubuntu_esxi2 (docker_slave_ssh) in workspace /home/jenkins/workspace/my_python_job The recommended git tool is: NONE No credentials specified Cloning the remote Git repository Cloning repository https://github.com/yourrepojenkins-docker-slave > git init /home/jenkins/workspace/my_python_job # timeout=10 Fetching upstream changes from https://github.com/yourrepojenkins-docker-slave > git --version # timeout=10 > git --version # 'git version 2.34.1' > git fetch --tags --force --progress -- https://github.com/yourrepojenkins-docker-slave +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/yourrepojenkins-docker-slave # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch > git rev-parse refs/remotes/origin/master^{commit} # timeout=10 Checking out Revision f0943b9194a48bafebf77280e1a848af92c6d6c3 (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f f0943b9194a48bafebf77280e1a848af92c6d6c3 # timeout=10 Commit message: "QA: adds ! to helloworld.py" > git rev-list --no-walk 09810fbbf97a812d396aca02e50c443ce999fa8f # timeout=10 [my_python_job] $ /bin/sh -xe /tmp/jenkins8662150179336980601.sh + python3 helloworld.py Hello World! Finished: SUCCESS

If your job is a Multibranch Pipeline, you can use

Configure → Scan Multibranch Pipeline Triggers → Periodically if not otherwise run → Interval

Where you can set the desired interval.

This feature can work together with webhooks from a remote repository, and is usually used as a fallback in case a webhook did not fire.

Environment variables

You can study the available environment variables on the page

http://jenkins_url/env-vars.html

For example, in my case it is

http://jenkins.andrei.com:8080/env-vars.html/

Credentials

By installing the Credentials plugin you can centrally store the login, password and other authentication means right in Jenkins

Credential Scopes:

System - available only on the Jenkins server and nodes. They are not visible to Jenkins jobs

Global - available to everyone.

If you go to a Pipeline, there will also be a Credentials section: there you set Credentials local to this project and not visible from others. This requires the Folders plugin.

Creating Jenkins Credentials, image from www.aredel.com
Click Add Credentials
www.aredel.com
Creating Jenkins Credentials, image from www.aredel.com
Creating Credentials
www.aredel.com

It is important not to forget to specify the ID, since it is what identifies the data. If you do not set it manually, Jenkins will generate an ID itself and it will be harder to remember.

To use the credentials stored in Jenkins, you need to install the plugin Credentials Binding Plugin

Credentials Bind Jenkins, image from www.aredel.com
Plugin
www.aredel.com

For how to use credentials in a Pipeline, read the article «Using credentials in Jenkins Pipeline»

Project builders

pipeline { agent any tools { gradle 'Gradle-6.2' } stages { stage("run frontend") { steps { echo 'executing yarn…' nodejs('Node-10.17') { sh 'yarn install' } } } stage("run backend") { steps { echo 'executing gradle…' sh './gradlew -v' } } } }

Article author: Andrei Olegovich