CI CD pipeline for a Python project using Git, Docker, Jenkins and Ansible

Submitted on Wed, 06/26/2024 - 20:23

In this tutorial, we will build a CI CD pipeline for a Python Flask project using Git, Docker, Jenkins and Ansible. Git shall be used for version control. Docker shall be used for containerization. Jenkins shall be used for continuous integration including build automation and Ansible shall be used for continuous deployment.

Git and Docker shall be the prerequisites for this project implementation. Also, an account at hub.docker.com shall be required to host the docker images needed for this project.

1. Prepare a python project

app.py
pyreq.txt
Dockerfile

app.py
from flask import Flask
app=Flask(__name__)
@app.route('/')
def webout():
    return '<h1>DevOps is fun.</h1>'
app.run(host='0.0.0.0',port=7000)

 

pyreq.txt
Flask==2.0.3
Werkzeug==2.0.3

 

Dockerfile
FROM python:3.9-slim
COPY . /app
WORKDIR /app
RUN pip install --no-cache-dir -r pyreq.txt
EXPOSE 7000
CMD ["python","app.py"]

 

2. Build the Docker image and run a container

docker build -t apurwasingh/flask .
docker run -dit -p 8000:7000 --name flask apurwasingh/flask

Note: Port 8000 on the host OS is mapped to port 7000 on the container.

Make sure that port 8000 on the local machine is available. 

Ports in use can be found with following command:

On Unix like OS:

netstat -tlpn

On windows:

netstat -aon | findstr LISTENING

On macOS:

netstat -anv | grep LISTEN

Make sure to use ports which are not in use yet.

After this command, go to localhost:8000 and see the web application in action.

 

3. Key based authentication

Generate ssh key pair in local machine

ssh-keygen -t rsa -b 4096 -C "apurwa@gmail.com"

Start the SSH agent with following command:

eval "$(ssh-agent -s)"

Add the ssh key to the ssh agent:

ssh-add ~/.ssh/id_rsa

 

Note: For Windows, the ssh agent can be started from powershell with following command:

Start-Service ssh-agent

If the agent cant be started, go to Windows Services and check for a service called OpenSSH Authentication Agent. Make sure that it is started. Once that is done, try the command again. After that, add the ssh keys to the agent with following command

ssh-add <path to private key>

 

Add public key to github

Go to github.com/settings/keys, create a new SSH key and paste the cotent of the public key id_rsa.pub.

Test connection from the host OS as follows:

ssh git@github.com

Note: Don't use your username

You should see a message like this:

Hi apurwa-np! You've successfully authenticated, but GitHub does not provide shell access.

Connection to github.com closed.

Now, we can push to github repo using key based authentication from our host OS.

 

4. Prepare a GitHub repo for the project

Create a new repo at github (devops in this case)

Copy the SSH version of the repo’s URL.

(git@github.com:apurwa-np/devops.git)

Go to the folder containing the python project in the local machine.

Initialize git with

git init

 

Configure git

git config user.email apurwa@gmail.com

git config user.name apurwa

git remote add heavens git@github.com:apurwa-np/devops.git

 

Push the source code to github

git add .

git commit -m "first commit"

git push heavens main

Note: Here, our remote repo is called heavens.

 

5. Configure Jenkins server

Custom network configuration

Create a custom network for communication between Jenkins server and agents for running Jenkins pipeline

docker network create --subnet 10.10.10.0/24 jnet

 

Run Jenkins in docker container

docker run -d --name jenkins -p 8080:8080 -p 50000:50000 -v /var/run/docker.sock:/var/run/docker.sock -v jenkins_home:/var/jenkins_home --network jnet --ip 10.10.10.2 jenkins/jenkins:lts

Note: Here, we are mounting the docker socket file from host OS to the container. This will allow us to run docker commands from inside the container.

For Windows using Docker Desktop, the path to docker socket file is //var/run/docker.sock (notice an extra forward slash)

Hence, for Windows the command would be as follows:

docker run -d --name jenkins -p 8080:8080 -p 50000:50000 -v //var/run/docker.sock:/var/run/docker.sock -v jenkins_home:/var/jenkins_home --network jnet --ip 10.10.10.2 jenkins/jenkins:lts

Go to localhost:8080 and finish Jenkins setup. Jenkins will ask for an initial admin password. Run the following command in host OS to find it out:

docker exec -u root jenkins cat /var/jenkins_home/secrets/initialAdminPassword

 

Plugin configuration

After Jenkins is setup, go to Dashboard > Manage Jenkins > Plugins and install following plugin:

Docker pipeline

 

Host Key Verification configuration

Go to Manage Jenkins > Security

Scroll down to Git Host Key Verification Configuration.

Select Accept first connection as option for Host Key Verification Strategy.

Note: Without this, our host key verification will fail and our pipelines would not be able to use the ssh keys for authenticating to GitHub.

 

6. Configure Jenkins agent

Configure Jenkins Node

Go to Manage Jenkins > Nodes > New Node

Node name: docker-agent

Select Permanent Agent

Remote root directory: /home/jenkins

Labels: docked

In the Launch method, choose Launch agent by connecting it to the controller. In the next page, click on the newly created node.

A secret will be visible. Copy the secret alphanumeric value. It will be used as a JENKINS_SECRET variable with the docker run command as shown below:

 

Run a docker container for the agent with the secret and agent name

docker run -d --name jenkins-agent --restart=unless-stopped   -v /var/run/docker.sock:/var/run/docker.sock   -e JENKINS_URL=http://10.10.10.2:8080   -e JENKINS_SECRET=15c84ddc8a6a9a46c48cdf50449ed0b2623d10496ebb3e64a5a8caa2861fbd0c   -e JENKINS_AGENT_NAME=docker-agent   --network jnet --ip 10.10.10.3  jenkins/inbound-agent

 

For Windows,

docker run -d --name jenkins-agent --restart=unless-stopped   -v //var/run/docker.sock:/var/run/docker.sock   -e JENKINS_URL=http://10.10.10.2:8080   -e JENKINS_SECRET=15c84ddc8a6a9a46c48cdf50449ed0b2623d10496ebb3e64a5a8caa2861fbd0c   -e JENKINS_AGENT_NAME=docker-agent   --network jnet --ip 10.10.10.3  jenkins/inbound-agent

Note: Make sure that the JENKINS_AGENT_NAME is same as the node's name (docker-agent in this case)

 

Install docker and make the socket file writable by the jenkins user

docker exec -it -u root jenkins-agent bash

apt update && apt install -y docker.io

chown jenkins /var/run/docker.sock

Note: The jenkins user will run the pipeline. In order to execute docker commands, the socket file needs to be writable by the jenkins user.

 

7. Configure credentials

GitHub credentials

Manage Jenkins > Credentials

Click on global > Add credentials

Select SSH Username with private key

ID:  git-credentials

username: git

Private Key > Enter directly > Add

Paste the private key and click Create.

Note: For git, the username is always git.

 

Docker Hub credentials

Manage Jenkins > Credentials

Click on global > Add credentials

Select Username with password

Enter Username and password for the Docker Hub account.

Note: If gmail account is used for logging into Docker Hub, password is the gmail account's password. Docker Hub also uses personal access token mechanism which can be set here as password. For personal access token, go to hub.docker.com > Account Settings > Personal access tokens.

ID: docker-hub-credentials

Click on Create

 

8. Configure Ansible

Create a Dockerfile as follows:

FROM ubuntu:20.04

ENV ANSIBLE_VERSION 2.9.17

RUN apt update && apt install -y gcc python3 python3-pip vim && apt clean all

RUN pip3 install --upgrade pip && pip3 install ansible

 

Build Docker image and run a container

docker build -t ansible .

docker run -dit --name ansible --network jnet --ip 10.10.10.4  -v /var/run/docker.sock:/var/run/docker.sock  ansible

 

For Windows,

docker run -dit --name ansible --network jnet --ip 10.10.10.4  -v //var/run/docker.sock:/var/run/docker.sock  ansible

 

Install specific python modules

docker exec -it -u root ansible bash

pip3 install --force-reinstall 'requests<2.29.0' 'urllib3<2.0'

Note: Checkout this issue

pip3 install docker

Verify the python modules with following command:

pip3 list

 

Playbook configuration

Inside the ansible container, create a playbook at /root/deploy.yml as follows:

---

- name: Deploy the latest Flask app
  hosts: localhost
  tasks:

  - name: Stop the existing container
    docker_container:
      name: flask
      state: stopped
      image: apurwasingh/flask

  - name: Remove the existing container
    docker_container:
      name: flask
      state: absent

  - name: Pull the latest image
    docker_image:
      name: apurwasingh/flask
      tag: latest
      source: pull

  - name: Run the container with the latest image
    docker_container:
      name: flask
      image: apurwasingh/flask:latest
      state: started
      detach: yes
      tty: yes
      ports: [7000:7000]
      restart_policy: always

9. Configure Jenkins pipeline

New Item > Pipeline

Build Triggers: Poll SCM

Schedule: */5 * * * * (every 5 minutes)

Definition: Pipeline script from SCM

SCM: Git

Repository URL: git@github.com:apurwa-np/devops.git

 

Credentials: git

Branch Specifier: */main

Script Path: Jenkinsfile

Note: Please make sure you are using the correct branch name. In some OS, master branch is used while in some main is used.

 

10. Script configuration

Create a script file named Jenkinsfile in the project directory as follows:

pipeline{
agent { label 'docked' }

environment {
    DOCKERHUB_CREDENTIALS = credentials('docker-hub-credentials')
    DOCKER_IMAGE_NAME = 'apurwasingh/flask'
    DOCKER_IMAGE_TAG = 'latest'
    DOCKERHUB_REPO = 'apurwasingh/flask'

}

stages{
    stage('Checkout'){
        steps {
            checkout scm
        }
    }
    stage('Build'){
       steps{
            script{
                docker.build("${DOCKER_IMAGE_NAME}:${DOCKER_IMAGE_TAG}")
                docker.withRegistry('https://index.docker.io/v1/', 'docker-hub-credentials') {
                    docker.image("${DOCKER_IMAGE_NAME}:${DOCKER_IMAGE_TAG}").push()
                }
            }
        }
    }
    stage('Deploy'){
        steps{
            script{
                sh "docker exec -u root ansible ansible-playbook /root/deploy.yml"
            }
        }
    }
}

    
}

 

Push the update to Github including the Jenkinsfile

Go to project directory

git add .

git commit -m "Jenkinsfile added"

git push heavens main

Done !

Now, every time a new commit is pushed to this GitHub repo, the Jenkins pipeline will be executed. Based on the script, the pipeline shall do the following:

  1. Checkout the latest source code from GitHub.
  2. Build a new docker image based on the latest source code.
  3. Push the image to Docker hub repository.
  4. Remove the running container.
  5. Run a new container based on the new image.