CI/CD

What is CI/CD?

CI/CD stands for Continuous Integration and Continuous Delivery (or Continuous Deployment). It's a set of practices that automate the process of integrating code changes, testing them, and deploying them to production.

Think of CI/CD as an automated assembly line for software. Instead of manually checking code, running tests, and deploying - everything happens automatically whenever you push code changes.

Understanding the Components

Continuous Integration (CI)

CI is the practice of frequently merging code changes into a shared repository. Each merge triggers an automated build and test process.

• Frequent commits: Developers push code multiple times per day
• Automated builds: Code is compiled/packaged automatically
• Automated tests: Unit tests, integration tests run on every push
• Fast feedback: Developers know within minutes if something broke

Continuous Delivery (CD)

CD extends CI by automatically preparing code for release. After passing all tests, code is ready to be deployed to production at any time with a single click.

• Always deployable: The main branch is always in a deployable state
• Manual approval: A human decides when to actually deploy
• Staging environments: Code is tested in production-like environments

Continuous Deployment

Takes CD one step further - every change that passes all tests is automatically deployed to production. No human intervention required.

• Fully automated: Push code → automatically in production
• Requires confidence: Your tests must be comprehensive
• Small changes: Deploy many small changes rather than big releases

The CI/CD Pipeline

A CI/CD pipeline is the automated workflow that takes code from commit to production:

Developer pushes code
        ↓
    [Source Stage]
    - Code is pulled from repository
        ↓
    [Build Stage]
    - Dependencies installed
    - Code compiled/packaged
        ↓
    [Test Stage]
    - Unit tests
    - Integration tests
    - Code quality checks
        ↓
    [Deploy to Staging]
    - Deploy to test environment
    - Run end-to-end tests
        ↓
    [Deploy to Production]
    - Deploy to live servers
    - Monitor for issues

Why CI/CD Matters

• Catch bugs early: Find issues immediately after they're introduced
• Reduce risk: Small, frequent deployments are easier to fix than big releases
• Faster delivery: Get features to users quickly
• Developer productivity: Automate repetitive tasks
• Consistent quality: Every change goes through the same checks
• Confidence: Trust that your code works because it's tested

CI/CD for Python Projects

Here's how CI/CD typically looks for a Python application:

Project Structure

my-python-app/
├── app/
│   ├── __init__.py
│   ├── main.py
│   └── models.py
├── tests/
│   ├── __init__.py
│   ├── test_main.py
│   └── test_models.py
├── requirements.txt
├── requirements-dev.txt
├── Dockerfile
├── .github/
│   └── workflows/
│       └── ci.yml          # CI/CD pipeline definition
└── README.md

Example GitHub Actions Pipeline

# .github/workflows/ci.yml
name: CI/CD Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: '3.11'

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        pip install -r requirements-dev.txt

    - name: Run linting
      run: |
        pip install flake8
        flake8 app/ tests/

    - name: Run tests
      run: |
        pytest tests/ --cov=app --cov-report=xml

    - name: Upload coverage
      uses: codecov/codecov-action@v3

  build:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'

    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Build Docker image
      run: docker build -t myapp:${{ github.sha }} .

    - name: Push to registry
      run: |
        docker tag myapp:${{ github.sha }} registry.example.com/myapp:latest
        docker push registry.example.com/myapp:latest

  deploy:
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'

    steps:
    - name: Deploy to production
      run: |
        # SSH to server and pull new image
        ssh user@server "docker pull registry.example.com/myapp:latest && docker-compose up -d"

Types of Tests in CI/CD

Unit Tests

Test individual functions or classes in isolation. Fast and run on every commit.

# tests/test_calculator.py
import pytest
from app.calculator import add, multiply

def test_add():
    assert add(2, 3) == 5
    assert add(-1, 1) == 0

def test_multiply():
    assert multiply(3, 4) == 12
    assert multiply(0, 5) == 0

Integration Tests

Test how different parts of your application work together.

# tests/test_api.py
import pytest
from fastapi.testclient import TestClient
from app.main import app

client = TestClient(app)

def test_create_user():
    response = client.post("/users/", json={
        "username": "testuser",
        "email": "test@example.com"
    })
    assert response.status_code == 201
    assert response.json()["username"] == "testuser"

def test_get_user():
    response = client.get("/users/1")
    assert response.status_code == 200

End-to-End Tests

Test the entire application flow from user interface to database.

Code Quality Checks

• Linting: Check code style (flake8, pylint)
• Type checking: Verify type hints (mypy)
• Security scanning: Find vulnerabilities (bandit)
• Coverage: Measure test coverage

Popular CI/CD Tools

GitHub Actions

Built into GitHub, great for most projects. Free for public repositories.

GitLab CI

Integrated with GitLab. Powerful and feature-rich.

Jenkins

Open-source, self-hosted. Very flexible but requires more setup.

CircleCI

Cloud-based, fast builds, good Docker support.

Travis CI

Simple configuration, popular with open-source projects.

CI/CD Best Practices

• Keep pipelines fast: Aim for under 10 minutes. Slow pipelines kill productivity.
• Fail fast: Run quick tests first to catch obvious errors early.
• Make builds reproducible: Same code should always produce the same result.
• Version your infrastructure: Pipeline configuration should be in version control.
• Use feature branches: Test changes before merging to main.
• Automate everything: If you do it manually more than twice, automate it.
• Monitor deployments: Know immediately when something goes wrong.
• Have rollback plans: Be able to quickly revert bad deployments.

A Simple CI/CD Setup

Start simple and add complexity as needed:

# .github/workflows/simple-ci.yml
name: Simple CI

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: '3.11'

    - name: Install dependencies
      run: pip install -r requirements.txt

    - name: Run tests
      run: pytest

# That's it! Start here and add more steps as needed.

Common CI/CD Challenges

• Flaky tests: Tests that sometimes pass and sometimes fail. Fix them immediately.
• Slow pipelines: Parallelize tests, cache dependencies, optimize Docker builds.
• Environment differences: Use containers to ensure consistency.
• Secret management: Never commit secrets. Use environment variables or secret managers.
• Database testing: Use test databases or mocks, not production data.

Related Articles