How to Setup Docker Containers as Build Agents for Jenkins

Docker containers as Build Slaves for Jenkins

This article guides you to set up Jenkins build inside a Docker container using Docker-based Jenkins build agents.

The resource utilization of the Jenkins agents is very less if you do not have builds happening continuously. It is better to use ephemeral Docker containers as Jenkins agents for better resource utilization in this scenario.

As you know, spinning up a new container takes less than a minute; every build spins up a new container, builds the project, and is destroyed. This way, you can reduce the number of static Jenkins build VMs.

Docker Containers as Build Agents/Slaves

In this guide, I will walk you through the steps for configuring Docker Containers as build agents.

I assume that you have a Jenkins server up and running. If you do not have one, follow this tutorial. How to setup Jenkins 2

If you want docker based Jenkins setup, you can follow this tutorial -> Setup Jenkins On a Docker container

Let’s Implement It

Configure a Docker Host With Remote API [Important]

The first thing we should do is set up a docker host. Jenkins server will connect to this host for spinning up the build agent containers. I am going to use the Centos server as my docker host. You can use any OS which supports Docker.

Jenkins master connects to the docker host using REST APIs. So we need to enable the remote API for our docker host.

jenkins docker slave

Make sure the following ports are enabled in your server firewall to accept connections from Jenkins master.

Docker Remote API port4243
Docker Hostport Range32768 to 60999

32768 to 60999 is used by Docker to assign a host port for Jenkins to connect to the container. Without this connection, the build slave would go in a pending state.

Lets get started,

Step 1: Spin up a VM, and install docker on it. You can follow the official documentation for installing docker. based on the Linux distribution you use. Make sure the docker service is up and running.

Step 2: Log in to the server and open the docker service file /lib/systemd/system/docker.service. Search for ExecStart and replace that line with the following.

ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock

Step 3: Reload and restart docker service.

sudo systemctl daemon-reload
sudo service docker restart

Step 4: Validate API by executing the following curl commands. Replace 54.221.134.7 with your host IP.

curl http://localhost:4243/version
curl http://54.221.134.7:4243/version

Check the docker remote API article for a detailed explanation of Docker API.

Once you enabled and tested the API, you can now start building the docker slave image.

Create a Jenkins Agent Docker Image

I have created a Jenkins docker image for maven. You can use this image or use its Dockerfile as a reference for creating your own.

If you are creating the image on your own, its image should contain the following minimum configurations to act as a slave.

  1. sshd service running on port 22.
  2. Jenkins user with password.
  3. All the required application dependencies for the build. For example, for a java maven project, you need to have git, java, and maven installed on the image.

Make sure the sshd service is running and can be logged into the containers using a username and password. Otherwise, Jenkins will not be able to start the build process.

Note: The default ssh username is jenkins and the password is also jenkins as per the given Dockerfile. You will have to use these credentials in the below configuration.

Configure Jenkins Server With Docker Plugin

Step 1: Head over to Jenkins Dashboard –> Manage Jenkins –> Manage Plugins.

Step 2: Under the Available tab, search for “Docker” and install the docker cloud plugin and restart Jenkins. Here is the official plugin site. Make sure you install the right plugin as shown below.

Install Jenkins Docker Plugin

Step 3: Once installed, head over to Jenkins Dashboard –> Manage Jenkins –>Configure system.

Step 4: Under “Configure System“, if you scroll down, there will be a section named “cloud” at the last. There you can fill out the docker host parameters for spinning up the slaves.

Note: In Jenkins versions 2.200 or later you will find dedicated cloud configuration under Manage Jenkins –> Manage Nodes and Clouds

Step 5: Under docker, you need to fill out the details as shown in the image below.

Note: Replace “Docker URI” with your docker host IP. For example, tcp://10.128.0.3:4243 You can use the “Test connection” to test if Jenkins is able to connect to the Docker host.

configure Jenkins docker cloud

Step 6: Now, from “Docker Agent Template” dropdown, click the “Add Docker template” and fill in the details based on the explanation and the image given below and save the configuration.

  1. Labels – Identification for the docker host. It will be used in the Job configuration. Here we use java-docker-slave
  2. Name: Name of the docker template. Here we use the same name as label ie, java-docker-slave
  3. Docker Image bibinwilson/jenkins-slave:latest or the image that you created for the slave.
  4. Remote Filing System Root – Home folder for the user you have created. In our case, it’s /home/jenkins
  5. Credentials – click add and enter the SSH username and password that you have created for the docker image. Leave the rest of the configuration as shown in the image below and click save. If you are using my Docker image, the user will be jenkins & password is also jenkins.

Note: There are additional configurations like registry authentication and container settings that you might have to use when configuring this set up in the corporate network.

Jenkins Docker agent tempalte configuration

You can also use JNLP-based slave agents. For this, the configurations need a little change as shown below. Primarily the docker image name and the connect method.

Note: For JNLP to work, you need to enable the JNLP connection port (50000) in Jenkins’s global security configuration (TCP port for inbound agents). Also, the Jenkins master firewall should be able to accept this connection form the docker host.

Docker jnlp agent template configuration

By default, the workspace will not be persisted in the host. However, if you want the workspace to be persistent, add a host volume path under container settings.

For example, if you want the workspace to be available at /home/ubuntu, you can add the volume path as shown below. /home/jenkins is the path inside the container.

/home/ubuntu:/home/jenkins

Towards the right of the Volumes option, if you click the question mark, it will show you additional volume options as shown below.

mount volume to Jenkins docker agent

Building Docker Images using Docker Agents

There are use cases where you have to build Docker images in the CI process.

You can achieve that using Docker based Jenkins agent as well.

If you are planning to run docker in docker for your CI process, you can mount the host docker.sock as volume to execute docker commands.

Check out my article on running docker in docker to know how to run docker in docker.

Test Jenkins Build Inside a Docker container

Now that you have the slave configurations ready, we will test the docker agent plugin using a freestyle job.

  1. Create a freestyle job, select “Restrict where this project can be run” option and select the docker host as a slave using the label.
  2. Add a shell build step which echoes a simple “Hello World
Jenkins build job with container as build agent

If you have done all the configurations right, Jenkins will spin up a container, builds the project, and destroys the container once the build is done.

First, you will see a pending notification as Jenkins tries to deploy a container on run time and establishes an SSH connection. After a few seconds, your job will start building.

image

You can check the build logs in your jobs console output as shown below.

docker slave output min

Also, you can check out the video explaining the whole process.

Possible Errors:

  1. Jenkins is not able to deploy containers on the host:– Please make sure you have proper connectivity to the docker host on API port.
  2. Jenkins builds goes in the pending state forever:– Make sure you have Docker host ports (32768 to 60999) access from Jenkins master to docker host.
  3. JNLP slaves go into the pending state: Make sure you have enabled the JNLP port in the Jenkins global security configuration.

Conclusion

In this article, I walked you through the process of setting up dynamic Jenkins Docker agents

If you want to build Jenkins inside a Docker container, and you don’t have a Kubernetes setup, this is the best option.

If you have a Kubernetes setup, then you consider setting up the agents on Kubernetes. You can check my blog on Kubernetes based Jenkins build agents to for more details.

It can be further customized to fit your specific use cases.

Please let me know your thoughts in the comment section. Also, don’t forget to share this article 🙂

67 Shares:
97 comments
  1. I have multiple cloud templates configured(with slightly different container configurations) that have the same Label. How do you set a preference? I added the “Preference of Node” attribute to each one and set the scores… but jenkins just ignores those. No matter how I have the scores set, it uses the first template configured with that label till it hits the max number of instances I set then it moves to the next one. any help would be appreciated.

  2. Hi Bibin,

    This is a great article.! I have tried the above steps but I am getting (pending—All nodes of label ‘build1-linux’ are offline). However, I am able to ssh to the docker container. Do you know what could be the issue here?

    1. Hi Suresh, Please check the firewall connectivity to Jenkins master from the agent node for ports 32768 to 60999

      1. Hi there,
        If you are running jenkins on docker, make sure you exposed the range port.
        $ docker run -d –expose=32768-60999 …..

  3. Hello,

    I tried to use JNLP and I followed all steps mentioned above. I also configured TCP to 50000.

    My problem is that JNLP keeps restarting it keeps making nodes is there any solution of this?

      1. My port range is enabled as the ssh docker connection worked. However, when it comes to JNLP, an agent.jar is needed to spin up a connection between Jenkins and the Docker Container. Error in Jenkins:

        This agent is offline because Jenkins failed to launch the agent process on it.

  4. It was a very good structure, I was trying to adopt it here but my main problema (and everyone’s else, as far as I have researched) is about concurrent builds.
    As all of those containers are tied to the same label this prevent to run more than one job at once.
    Any tips to workaround it ?

    1. You can run concurrent builds with the same label. It depends on the docker plugin settings where you mention the number of containers it is allowed to run concurrently.

  5. This is really good for testing out jenkins agent in docker form.
    I have 2 questions regarding its practical value:
    1. seems all docker container will be running on the same VM (docker host), correct? If so if the VM resource is limited then it can’t really scale out
    2. jenkins agent already support multiple “executors”, so the node can support multiple executors already?
    Thanks
    -cl

    1. 1. if you want to scale agent with the container, you should consider using Kubernetes based distributed agents.
      2. Even though Jenkins supports executors, Docker-based agents will give you isolated environments for each build. For example, different java versions, python versions, other utilities. In some cases, we have to have different agent VMs for different use cases. We can avoid it by having dedicated container images for specific build use cases.

  6. Hi Bibin:
    I just want to know if i can commit the docker as an new image after the building on the docker slave.
    I tried it and found that the plugin destroy my docker once the build done.

    1. Hi Wzou,

      The build container is ephemeral in nature by design. So it won’t persist any data. You might consider normal docker based builds with shared workspaces concept if you want to persist data after building in a docker image.

  7. Thanks for the very informative article. My team is experimenting with a PaaS platform. They offer a VM that is configured as a Jenkins node. I’m just wondering if it’d be possible to deploy and run the container built as part of the Jenkins pipeline on the VM jenkins node. Please let me know.

    1. Yes! in the VM you need to set up Docker on the VM node and in your pipeline you can say which configure the image to run.

  8. Hi Bibin,
    Thanks for such a wonderful article and very precisely explained. I have done the similar configurations and able to connect docker containers as jenkins on-demand slave.

    I have a query here.
    Is there a way to use DNS hostname instead of tcp IP in Docker URI ?
    The idea is to use a load balancer under which multiple instances can be attached.

    Regards
    Prakash K

    1. Hi Prakash,

      We are not sure about DNS. Please give it a try and let us know. Logically it should work

  9. hi,
    i am not able to build the job in docker it keeps the job in queue
    it shows “(pending—All nodes of label ‘docker-slave’ are offline)” even if the container is running and ssh service is started

    1. Can you check the firewall & connetivity between jenkins and docker host? Also, JLP port connectivity from docker host to jenkins machine

  10. Hello,

    Thanks for the page. Its really Helpful. But, if the container is removed then what is the use of it?How can we check if the build was successful? I want to deploy my webserver on dockerhost & browse through public IP.

    How to achieve this?

    Thanks in advance..

    1. amazing article and well defined every an armature can also follow this.
      but how to configure so that it connects host jenkins network or other tools hosted in organization.

      1. Hi Amir. Glad it helped.

        If your organization has proxy and other network configurations, you need to configure those settings on the based docker slave image. In most cases, you will have to build your own slave image with network, proxy, DNS configurations from the organization. Once you do this the docker slave will behave like any other server in your infrastructure.

        1. Thanks for your reply,

          I am trying it to test it on AWS EC2 instance. I dont have any proxy or other secured network configuration. My image is a basic httpd image where i am trying to mount a local ebs backed volume mapped to httpd document root & trying to look that over web browser.

          But i have noticed that the containers are removed immediately as its created. I have also made sure that it has one process running while creation & it works in normal docker server very efficiently.

          Thanks in advance.

    1. @arun. thanks for your feedback. We have asked the author to make the necessary changes and it is rectified now.

  11. Hi,
    This is a best document so far for running jenkins docker slaves.
    But I am facing an issue.
    The docker container doesn’t spin up.
    My Settings are:
    Connect method: Connect with SSH
    SSH Key: Use configured SSH credentials
    Host Key Verification Strategy: Non verifying
    But the docker slave doesn’t start, it remain in pending status and get below error message in the docker template under cloud in configure system:
    Reason: Template provisioning failed.
    java.lang.NullPointerException
    io.jenkins.docker.connector.DockerComputerSSHConnector.getBindingForPort(DockerComputerSSHConnector.java:293) at io.jenkins.docker.connector.DockerComputerSSHConnector.createLauncher(DockerComputerSSHConnector.java:245) at io.jenkins.docker.connector.DockerComputerConnector.createLauncher(DockerComputerConnector.java:91) at

  12. how to set Docker Host IP in step-5.
    i’m using below command to run docker image:
    docker run -d -p 4243:22 –name udockerslave bibinwilson/jenkins-slave:latest
    it binds docker at 0.0.0.0:4243,i can use -p 127.0.0.1:4243:22 but can’t use any other private ip like -p 192.168.33.10:4243:22, please suggest!

  13. how to run this docker container, i’m trying command:
    docker run -d -p 4342:22 –name bibinwilson/jenkins-slave:latest
    giving me error like:
    “docker run” requires at least 1 argument.
    i was trying to use 4342 as host machine port and 22 as container port, bcz ssh 22 port is used for the host machine,pls help!

    1. You need to provide a name for your container.

      docker run -d -p 4342:22 –name bibinwilson/jenkins-slave:latest

  14. I do not understand how this method is “better resource utilization” more then setting up a single slave and run against it all the builds.
    Why do I need a docker setup?

  15. i follwed same steps as you did but no success. problem with ssh connection. i installed jenkins ad docker on same machine. the new version of docker plugin is not showing docker ssh computer launcher

  16. What happens to the artefacts ( build artefacts that are generated ) before terminating the docker ?

    Do we really need a docker container just for spinning a build and then killing or terminating the container . I understand the use case and appreciate the use of dockers for it but not able to see the advantage of the same.

  17. Hello Nice Articale..Just had 1 question as what will happen to workspace after the slave are terminated after the build. Am using K8s plugin for spinning up the slaves and it works the same way as Docker. But once the slave terminate I am no more able to access the workspace as slave node is terminated. I tried NFS option but it didn’t help. So any such issue if you have face. Really appreciate some pointers on this query

  18. When I kickoff a job in Jenkins which is configured to run on a Docker Slave, I get the following ERROR

    [07/31/18 16:51:20] [SSH] Opening SSH connection to HOST.
    [07/31/18 16:51:20] [SSH] WARNING: SSH Host Keys are not being verified. Man-
    in-the-middle attacks may be possible against this connection.
    ERROR: Failed to authenticate as admin. Wrong password.
    (credentialId:43b12130-032d-48df-8724-6588e9a472a9/method:password)
    [07/31/18 16:51:22] [SSH] Authentication failed.
    Authentication failed.
    [07/31/18 16:51:22] Launch failed – cleaning up connection
    [07/31/18 16:51:22] [SSH] Connection closed.

    Where can I specify the credentials for logging in to the container??

  19. I’m also getting #1 (pending—Jenkins doesn’t have label docker-slave) as someone else mentioned. Is there any reason why?

  20. I created a build after above set up. But it always says:

    #1
    (pending—Waiting for next available executor)

    It is not getting the agent to run I believe. Any hints?

      1. Please check the firewall connectivity between Jenkins and the docker host. Also, check Jenkins logs and see whats happening

  21. Hi, thank you for the post.
    I have one question, how can I use the “Restrict where this project can be run” option if i’m using Jenkinsfile?
    I want to run a job that is a scripted pipeline in a docker-slave.
    Thank you

  22. What if I have application teams which use different languages viz. one Java and the other Go ? In such a situation should I create multiple images for each type of application and then further add them in Jenkins server configuration ? How can we separate whenever a job starts, how do we make sure a Go-lang application only gets built by the appropriate image.

  23. Please let me know how to setup to let slave able to run docker command, I tried with mouting docker.sock and docker binary on Container Mount Points of ECS slave templates (Manage Jenkins -> Config System -> Cloud -> Amazon EC2 Container Service Cloud -> ECS slave templates -> ADVANCED -> Container Mount Points) does not work.

  24. Using this plugin when i start a new build it is taking more time to clone the git repository i chose git potion for source code management.
    Is there any docker cache not to take more time for the same sources

  25. Hi Arnold,

    I just found the docker containers node can be saved. From manage jenkins. –>. configure system –> Cloud —> docker template —> Availability, select “Experimental: Keep this agent online as much as possible”.

  26. Hi All,
    I am running multiple docker servers running 300+ jobs. If one server goes down i have observed all other servers will go offline. Any idea on this?

  27. Helpful Article !!. Bibin, Can you make an article how to upgrade Jenkins which is in HA cluster. I tried to do some research but I couldn’t find any articles about this. You are making a crystal clear overview , it can help lots of people

  28. Hi Bibin,
    If this does work will the slave appear under the Jenkins nodes on the master?

    Thanks.

    1. Hi Arnold,

      Docker containers will get spun up on the go and gets destroyed after the build. So I don’t think it will be listed in the node list.

  29. Hi, I looked @ your slave docker image, would you mind sharing the dockerfile that created it?

  30. Hi Bibin,

    How can I build pipeline jobs on Docker Slaves once this kind of job there’s no “Restrict where this project can be run” option to select and set the label restriction? In my environment I have physical slaves and docker slaves, but without that option, all jobs run in the physical slave.

  31. Thanks for this blog, it helped me setup my Jenkins-Docker environment. Though I am trying to figure out a way to retain the containers if the build fails. The idea is to troubleshoot the failure. Is there a way to do it ?

  32. This works but I guess I assumed if a second job was triggered while a job is currently running, it would run a new slave. What I actually see is a single slave is started, if a second build is triggered while the first is still running, it tries to wait for/run on the same slave. When the first build finishes, the slave is torn down, the second job restarts on a different slave and runs.

    Is there anyway to run multiple slaves on the same host?

    1. Hi dtothefp,

      Iam trying very hard to fix this but completely in vain every time. My issue is
      In my Vm iam using centos 7 and have launched docker container running jenkins master.
      in jenkins iam using docker plugin , Now i need the docker url to connect to.

      Can u pls help this issue…

      Thanks

  33. i followed same as given and when trigger the build showing
    ” #1 (pending—Jenkins doesn’t have label docker-slave) “.

    Please guide me regarding this.

  34. This is the exact setup that I have, which works pretty well. It would be great to show a tutorial on how to have this run in a docker swarm setup (so basically same thing but show how to properly configure the docker swarm to make it work properly with Jenkins).

  35. hi,

    i created a docker image but jenkins is not able to log into that container image that showing
    “pending—All nodes of label ‘docker-slave’ are offline)” onjenkins log

    pleas help me

  36. Thanks for the post. I’m curious of how this works with a workflow where the repos you want to test/build in CI are actually managed by docker. I’ve messed around on a jenkins-master that I scaled up using Docker, and exposed the Docker socket on the host to the CI container, allowing me to run docker commands inside my builds that were running on jenkins-master.

    Now that I’m running builds inside Docker containers on a remote host I’m wondering what the preferred method for doing this would be. I’ve seen explanations of Docker-in-Docker vs. Docker-outside-of-Docker, and I’m unsure what is the better approach? I’m assuming that my Docker container that is equivalent of bibinwilson/jenkins-slave could either expose the Docker socket to the CI build or could somehow install Docker inside the container. Are there best practices here or am I thinking about it all wrong?

  37. Pingback: How to Setup Docker containers as Build Slaves for Jenkins – DevOps Infographics Resource Center – Programming and IT
  38. Hi Bibin,

    with one colleague of mine we put in place a similar process, the only issue for me is to connect to the agent/container created by my job.
    When I look at the logs I can see that the connection is refused due the port used by jenkins is not matching the port that we binded with the port 22 exposed from the container.
    the strange part is that jenkins is incrementing the port used by 1 everytime a container is created.
    Does this ring a bell for you or do you have any suggestion?

    Many thanks,
    Darko

  39. Hi,

    I would be grateful to hear an answer for the following post.

    I have a docker running on my host, and have a container with jenkins running on it. There are other containers, say a1,a2,etc running parallely. Now, i would like to spin up a container (say C1 )for a build. C1 now has to spin up the containers a1,a2 within it. I understand that there must be a docker within C1. Can this be achieved using Jenkins docker plugin ?
    Also, i assume i need not expose any TCP socket, since i have jenkins running on the same host. Please explain the solution in detail.
    Thanks a ton,
    Thilak

  40. hi bibin,
    i have configured all as u mentioned above.
    but when i build the job its remians in queue.
    can u guide me in this???

  41. hi,
    i am not able to build the job in docker it keeps the job in queue
    it shows “(pending—All nodes of label ‘docker-slave’ are offline)” even if the container is running and ssh service is starded

  42. Hi Bibin,

    I already tried this, this step is creating the image, but not spinning the containers.

    Thanks,
    Prasenjit

    1. Hi Nancy,

      The plugin is made that way to destroy the containers. You can commit the containers if you would like to persist them. Please go the plugin documentation. There it it mentioned clearly.

  43. I am able to create the image now, but the container is not spinning with any of the plugins, could you please let me know which plugin to be used to spin the container?

  44. Bibin,

    I am able to create the image now, but the container is not spinning with plugins, could you please let me know which plugin to be used to spin the container?

    Thanks,
    Prasenjit

  45. Hi Bibin,

    I tried all possible ways, when after configuration i start the docker as process, it gets connected with the jenkin, but the docker daemon thread doesn’t get up and thus any docker command dosn’t executes.

    But when restarts the docker machine as daemon, the changes doesnot reflects and thus connectivity fails.

    Could you please provide some input on this.

    Thanks,
    Prasenjit

  46. How to get the docker url….i tried with http://:2375, but connection is failing…..could you please suggest, whether wrong with the port number? if so , please let me know how can i get the port number

  47. Hi Bibin, will it auto create the slave and destroy it after the build. What if the docker slave is created successfully and the build fails (Maven/Gradle or any jenkins job related configuration/build/etc step fails? etc).

    1. HI Ramp,

      Yes. You can mount the jenkins workspace as a volume to the Docker slave. In your scripts running inside the container, you can write the logic to copy the compiled files before termination.

  48. Thank you for the great tutorial. I was wondering is it possible to archive artifacts built in Docker container in Jenkins? i.e. after the build completes, I go to Jenkins, click on the build run and download the build artifacts. Thanks!

    1. @disqus_sF63zwkuVN:disqus you can use nexus configuration to upload the artifacts to nexus. So after a successfull build, your artifact would automatically gets uploaded to nexus.

Leave a Reply

Your email address will not be published. Required fields are marked *

You May Also Like