In this tutorial, we will build a CI CD pipeline for a Python Flask project using Git, Docker, Jenkins and Ansible. Git shall be used for version control. Docker shall be used for containerization. Jenkins shall be used for continuous integration including build automation and Ansible shall be used for continuous deployment.
Git and Docker shall be the prerequisites for this project implementation. An account at hub.docker.com shall be required to host the docker images needed for this project. And finally, a GitHub account for hosting the source code of the project.
1. Prepare a python project
app.py
pyreq.txt
Dockerfile
app.py
from flask import Flask
app=Flask(__name__)
@app.route('/')
def webout():
return '<h1>DevOps is fun.</h1>'
app.run(host='0.0.0.0',port=7000)
pyreq.txt
Flask==2.0.3
Werkzeug==2.0.3
Dockerfile
FROM python:3.9-slim
COPY . /app
WORKDIR /app
RUN pip install --no-cache-dir -r pyreq.txt
EXPOSE 7000
CMD ["python","app.py"]
2. Build the Docker image and run a container
docker build -t apurwasingh/flask .
docker run -dit -p 8000:7000 --name flask apurwasingh/flask
Note: Port 8000 on the host OS is mapped to port 7000 on the container.
Make sure that port 8000 on the local machine is available.
Ports in use can be found with following command:
On Unix like OS:
netstat -tlpn
On windows:
netstat -aon | findstr LISTENING
On macOS:
netstat -anv | grep LISTEN
Make sure to use ports which are not in use yet.
After this command, go to localhost:8000 and see the web application in action.
3. Key based authentication
Generate ssh key pair in local machine
ssh-keygen -t rsa -b 4096 -C "apurwa@gmail.com"
Start the SSH agent with following command:
eval "$(ssh-agent -s)"
Add the ssh key to the ssh agent:
ssh-add ~/.ssh/id_rsa
Note: For Windows
The ssh agent can be started from PowerShell with following command:
Start-Service ssh-agent
If the agent cant be started, go to Windows Services and check for a service called OpenSSH Authentication Agent. Make sure that it is started. Once that is done, try the command again.
After that, add the ssh keys to the agent with following command
ssh-add <path to private key>
Add public key to github
Go to github.com/settings/keys, create a new SSH key and paste the content of the public key id_rsa.pub.
Test connection from the host OS as follows:
ssh git@github.com
Note: Don't use your username.
You should see a message like this:
Hi apurwa-np! You've successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed.
Now, we can push to GitHub repo using key based authentication from our host OS.
4. Prepare a GitHub repo for the project
Create a new repo at GitHub (devops in this case)
Copy the SSH version of the repo’s URL.
(git@github.com:apurwa-np/devops.git)
Go to the folder containing the python project in the local machine.
Initialize git with
git init
Configure git
git config user.email apurwa@gmail.com
git config user.name apurwa
git remote add heavens git@github.com:apurwa-np/devops.git
Note: Here, our remote repo is called heavens.
Push the source code to github
git add .
git commit -m "first commit"
git push heavens main
5. Configure Jenkins server
Custom network configuration
Create a custom network for communication between Jenkins server and agents for running Jenkins pipeline.
docker network create --subnet 10.10.10.0/24 jnet
Run Jenkins in docker container
docker run -d --name jenkins -p 8080:8080 -p 50000:50000 -v /var/run/docker.sock:/var/run/docker.sock -v jenkins_home:/var/jenkins_home --network jnet --ip 10.10.10.2 jenkins/jenkins:lts
Note: Here, we are mounting the docker socket file from host OS to the container. This will allow us to run docker commands from inside the container.
We also need to make sure that our docker socket file is available at the above mentioned path. In case of Docker Desktop, go to Settings > Advanced and tick the option Allow the default Docker socket to be used.
For Windows
The path to docker socket file is //var/run/docker.sock (notice an extra forward slash)
Hence, for Windows the command would be as follows:
docker run -d --name jenkins -p 8080:8080 -p 50000:50000 -v //var/run/docker.sock:/var/run/docker.sock -v jenkins_home:/var/jenkins_home --network jnet --ip 10.10.10.2 jenkins/jenkins:lts
Go to localhost:8080 and finish Jenkins setup. Jenkins will ask for an initial admin password. Run the following command in host OS to find it out:
docker exec -u root jenkins cat /var/jenkins_home/secrets/initialAdminPassword
Plugin configuration
During the installation, make sure to only select the following plugins to be installed:
- git
- pipeline
- credentials binding
- ssh agent
After Jenkins is setup, go to Dashboard > Manage Jenkins > Plugins and install following plugin:
Docker pipeline
These plugins are enough for our pipeline to run.
Host Key Verification configuration
Go to Manage Jenkins > Security
Scroll down to Git Host Key Verification Configuration.
Select Accept first connection as option for Host Key Verification Strategy.
Note: Without this, our host key verification will fail and our pipelines would not be able to use the ssh keys for authenticating to GitHub.
6. Configure Jenkins agent
Configure Jenkins Node
Go to Manage Jenkins > Nodes > New Node
Node name: docker-agent
Select Permanent Agent
Remote root directory: /home/jenkins
Labels: docked
In the Launch method, choose Launch agent by connecting it to the controller. In the next page, click on the newly created node.
A secret will be visible. Copy the secret alphanumeric value. It will be used as a JENKINS_SECRET variable with the docker run command as shown below:
Run a docker container for the agent with the secret and agent name
docker run -d --name jenkins-agent --restart=unless-stopped -v /var/run/docker.sock:/var/run/docker.sock -e JENKINS_URL=http://10.10.10.2:8080 -e JENKINS_SECRET=15c84ddc8a6a9a46c48cdf50449ed0b2623d10496ebb3e64a5a8caa2861fbd0c -e JENKINS_AGENT_NAME=docker-agent --network jnet --ip 10.10.10.3 jenkins/inbound-agent
For Windows
docker run -d --name jenkins-agent --restart=unless-stopped -v //var/run/docker.sock:/var/run/docker.sock -e JENKINS_URL=http://10.10.10.2:8080 -e JENKINS_SECRET=15c84ddc8a6a9a46c48cdf50449ed0b2623d10496ebb3e64a5a8caa2861fbd0c -e JENKINS_AGENT_NAME=docker-agent --network jnet --ip 10.10.10.3 jenkins/inbound-agent
Note: Make sure that the JENKINS_AGENT_NAME is same as the node's name (docker-agent in this case)
Install docker and make the socket file writable by the jenkins user
docker exec -it -u root jenkins-agent bash
apt update && apt install -y docker.io
chown jenkins /var/run/docker.sock
Note: The jenkins user will run the pipeline. In order to execute docker commands, the socket file needs to be writable by the jenkins user.
7. Configure credentials
GitHub credentials
Manage Jenkins > Credentials
Click on global > Add credentials
Select SSH Username with private key
ID: git-credentials
username: git
Private Key > Enter directly > Add
Paste the private key and click Create.
Note: For git, the username is always git.
Docker Hub credentials
Manage Jenkins > Credentials
Click on global > Add credentials
Select Username with password
Enter Username and password for the Docker Hub account.
Note: If a Gmail account is used for logging into Docker Hub, password is the Gmail account's password. Docker Hub also uses a personal access token mechanism which can be set here as password. For personal access token, go to hub.docker.com > Account Settings > Personal access tokens.
ID: docker-hub-credentials
Click on Create
8. Configure Ansible
Create a Dockerfile as follows:
FROM ubuntu:20.04
RUN apt update && apt install -y gcc python3 python3-pip vim && apt clean all
RUN pip3 install --upgrade pip && pip3 install ansible
Build Docker image and run a container
docker build -t ansible .
docker run -dit --name ansible --network jnet --ip 10.10.10.4 -v /var/run/docker.sock:/var/run/docker.sock ansible
For Windows
docker run -dit --name ansible --network jnet --ip 10.10.10.4 -v //var/run/docker.sock:/var/run/docker.sock ansible
Install specific python modules
docker exec -it -u root ansible bash
pip3 install --force-reinstall 'requests<2.29.0' 'urllib3<2.0'
pip3 install docker
Note: Here we are making sure that requests and urllib3 python modules are of specific versions. Checkout this issue
Verify the python modules with following command:
pip3 list
Playbook configuration
Inside the ansible container, create a playbook at /root/deploy.yml as follows:
---
- name: Deploy the latest Flask app
hosts: localhost
tasks:
- name: Stop the running container
docker_container:
name: flask
state: stopped
image: apurwasingh/flask
- name: Remove the stopped container
docker_container:
name: flask
state: absent
- name: Pull the latest image from docker hub
docker_image:
name: apurwasingh/flask
tag: latest
source: pull
- name: Run the container with the latest image
docker_container:
name: flask
image: apurwasingh/flask:latest
state: started
detach: yes
tty: yes
ports: [8000:7000]
restart_policy: always
9. Configure Jenkins pipeline
New Item > Pipeline
Build Triggers: Poll SCM
Schedule: */5 * * * * (every 5 minutes)
Definition: Pipeline script from SCM
SCM: Git
Repository URL: git@github.com:apurwa-np/devops.git
Credentials: git
Branch Specifier: */main
Script Path: Jenkinsfile
Note: Please make sure you are using the correct branch name. In some OS, master branch is used while in some main is used.
10. Script configuration
Create a script file named Jenkinsfile in the project directory as follows:
pipeline{
agent { label 'docked' }
environment {
DOCKERHUB_CREDENTIALS = credentials('docker-hub-credentials')
DOCKER_IMAGE_NAME = 'apurwasingh/flask'
DOCKER_IMAGE_TAG = 'latest'
DOCKERHUB_REPO = 'apurwasingh/flask'
}
stages{
stage('Checkout'){
steps {
checkout scm
}
}
stage('Build'){
steps{
script{
docker.build("${DOCKER_IMAGE_NAME}:${DOCKER_IMAGE_TAG}")
docker.withRegistry('https://index.docker.io/v1/', 'docker-hub-credentials') {
docker.image("${DOCKER_IMAGE_NAME}:${DOCKER_IMAGE_TAG}").push()
}
}
}
}
stage('Deploy'){
steps{
script{
sh "docker exec -u root ansible ansible-playbook /root/deploy.yml"
}
}
}
}
}
Push the update to Github including the Jenkinsfile.
Go to project directory
git add .
git commit -m "Jenkinsfile added"
git push heavens main
Done !
Now, every time a new commit is pushed to this GitHub repo, the Jenkins pipeline will be executed. Based on the script, the pipeline shall do the following:
- Checkout the latest source code from GitHub.
- Build a new docker image based on the latest source code.
- Push the image to Docker hub repository.
- Remove the running container.
- Run a new container based on the new image.