Machine learning and DevOps integration

5 min readMay 30, 2020

→First of all, many thanks to MR.VIMAL DAGA SIR for mentoring and training in Machine learning and DevOps from very basic to advance level

→Using this valuable knowledge I have completed the task of integration Machine learning with DevOps

→Tools used for DevOps — Git, Jenkins, Docker,

→Concept used for Machine Learning- Deep learning

SYSTEM CONFIGURATION/REQUIREMENTS-

→Base os Windows and virtual os Redhat Linux

→Jenkins installed in your RedHat system (including GitHub plugin)

→Docker installed with an image which can run python code

→Git installed in Windows and Redhat os

Problem statement- While creating a deep learning model, the process becomes too time taking and somewhat manual. And we all know, DevOps is used when we need automation!!

→Here deep learning will keep on creating the model and DevOps will keep on monitoring it and when deep learning gets accuracy lower than certain level Jenkins will automatically enhance the Dl code and will increase accuracy on its own

→STEP 1 →Creating a deep learning code and push it to GitHub with .PY file extension

→I have used cifar-10 dataset

Steps to get your code into GitHub-

1) git clone <repo. URL> — to clone the repository to your local system

2) ml_devops_integration1_py.py — code which is mentioned above

3) git add <filename> — to add the file into the staging area

4) git commit -m “message” — to make your changes permanent

5) git push origin — to push your changes into the remote repository

NOW HEADING TOWARDS CREATING A DOCKER IMAGE WHICH IS COMPATIBLE WITH PYTHON CODE

CREATE A DOCKERFILE WITH FOLLOWING

Run docker build command after creating docker file

→NOW THERE IS A PIPELINE OF 6 JENKINS JOBS

JOB1>This job will simply copy code from Github and paste into a specific folder in Redhat os

JOB2>This job is critical and important as well

→This container will first search for KERAS and it will launch the os with provided docker file so that only deep learning code runs on that os

Here job 1 is set upstream for this. Because it is important to get code from GitHub

Execute shell for reference-

“if sudo grep Keras | cat /MlOps_workspace/ml_devops_integration1_py.py
then
if sudo docker ps -a | grep img1
then
sudo docker rm -f img1
sudo docker run — name img1 -dit -v /MlOps_workspace:/Mlworkspace python_img:v1
else
sudo docker run — name img1 -dit -v /MlOps_workspace:/Mlworkspace python_img:v1
fi
else
echo “This is not a deep learning code”
fi”

→Here we have to mount permanent volume so that we can have our data persistent

JOB-3> This job will run the PYTHON code for initial level

Execute shell for reference-

sudo docker exec img1 python3 /Mlworkspace/ml_devops_integration1_py.py

JOB-4>>This job will check accuracy in our initial run

→If accuracy is less than 95% then Jenkins will do changes in code like increasing epochs and adding consolation layers to reach towards higher accuracy

Execute shell for reference-

read=$(sudo cat /MlOps_workspace/result1.txt)

echo $read*100 | bc -l
if (( $(echo “$read > 0.95” | bc -l) ));
then
echo “Achieved accuracy”
exit 1
else
cd /MlOps_workspace/
sudo sed -i ‘/^epoch=.*/a epoch=epoch+5’ ml_devops_integration1_py.py
sudo sed -i “62i model.add(Conv2D(filters=32, kernel_size=3, padding=’same’, activation=’relu’, input_shape=input_shape))” ml_devops_integration1_py.py
sudo sed -i “63i model.add(MaxPool2D(pool_size=2))” ml_devops_integration1_py.py
fi

→sed keyword is used to add specific code into our original file

JOB-5>This job will run again the edited code and find new accuracy