CI/CD Pipelines

Table of Contents

Overview

The CI/CD landscape in 2020 marked a significant shift toward container-native workflows, declarative pipeline definitions, and GitOps practices. This period saw the maturation of GitHub Actions, widespread adoption of GitLab CI, and the emergence of Kubernetes-native deployment tools like ArgoCD. Organizations moved from traditional Jenkins-based pipelines to more modern, cloud-native solutions that embraced infrastructure as code principles.

Background

The Evolution of CI/CD

Prior to 2020, Jenkins dominated the CI/CD space, but its plugin-based architecture and maintenance overhead led teams to seek alternatives. The containerization revolution with Docker and Kubernetes created demand for pipelines that could build, test, and deploy containerized applications seamlessly.

Key Market Trends

  • Shift from VM-based to container-based build environments
  • Adoption of YAML-based declarative pipeline definitions
  • Integration of security scanning (DevSecOps) into pipelines
  • Rise of GitOps as a deployment paradigm
  • Increased focus on developer experience and self-service platforms

Key Concepts

Continuous Integration (CI)

Continuous Integration focuses on automatically building and testing code changes as they are committed. Key practices include:

  • Automated builds triggered on every commit
  • Comprehensive test suites (unit, integration, end-to-end)
  • Static code analysis and linting
  • Security vulnerability scanning
  • Artifact generation and storage

Continuous Delivery/Deployment (CD)

Continuous Delivery ensures code is always in a deployable state, while Continuous Deployment automates the release to production:

  • Immutable artifacts (container images)
  • Environment promotion workflows (dev -> staging -> production)
  • Feature flags for gradual rollouts
  • Automated rollback capabilities
  • Blue-green and canary deployment strategies

GitOps

GitOps treats Git as the single source of truth for infrastructure and application deployments:

  • Declarative configuration stored in Git
  • Automated synchronization between Git state and cluster state
  • Pull-based deployments (cluster pulls changes from Git)
  • Audit trail through Git history
  • Self-healing infrastructure

Implementation

GitHub Actions

Released in 2019 and mature by 2020, GitHub Actions provides CI/CD directly integrated with GitHub repositories.

name: CI Pipeline
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Build Docker image
        run: docker build -t myapp:${{ github.sha }} .
      - name: Run tests
        run: docker run myapp:${{ github.sha }} npm test
      - name: Push to registry
        run: |
          echo ${{ secrets.DOCKER_PASSWORD }} | docker login -u ${{ secrets.DOCKER_USERNAME }} --password-stdin
          docker push myapp:${{ github.sha }}

Strengths:

  • Native GitHub integration
  • Extensive marketplace of reusable actions
  • Matrix builds for multi-platform testing
  • Self-hosted runners for custom environments

GitLab CI

GitLab CI offers a comprehensive DevOps platform with built-in CI/CD.

stages:
  - build
  - test
  - deploy

variables:
  DOCKER_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

build:
  stage: build
  image: docker:19.03
  services:
    - docker:19.03-dind
  script:
    - docker build -t $DOCKER_IMAGE .
    - docker push $DOCKER_IMAGE

test:
  stage: test
  image: $DOCKER_IMAGE
  script:
    - npm test
    - npm run lint

deploy-staging:
  stage: deploy
  script:
    - kubectl set image deployment/myapp myapp=$DOCKER_IMAGE
  environment:
    name: staging
  only:
    - main

Strengths:

  • Single platform for code, CI/CD, and container registry
  • Built-in security scanning (SAST, DAST, container scanning)
  • Review apps for merge request previews
  • Multi-project pipelines

CircleCI

CircleCI provides cloud-native CI/CD with strong parallelism support.

version: 2.1

orbs:
  docker: circleci/docker@1.0.0

workflows:
  build-and-deploy:
    jobs:
      - build
      - test:
          requires:
            - build
      - deploy:
          requires:
            - test
          filters:
            branches:
              only: main

jobs:
  build:
    docker:
      - image: cimg/node:14.0
    steps:
      - checkout
      - setup_remote_docker
      - run: docker build -t myapp .
      - docker/push:
          image: myapp

  test:
    docker:
      - image: cimg/node:14.0
    parallelism: 4
    steps:
      - checkout
      - run: npm ci
      - run: npm test -- --shard=$(echo $CIRCLE_NODE_INDEX)/$(echo $CIRCLE_NODE_TOTAL)

Strengths:

  • Excellent parallelism and test splitting
  • Orbs for reusable configuration
  • Insights and analytics
  • Resource classes for compute optimization

Jenkins

Jenkins remains relevant for organizations with complex, custom requirements.

pipeline {
    agent {
        kubernetes {
            yaml '''
                apiVersion: v1
                kind: Pod
                spec:
                  containers:
                  - name: docker
                    image: docker:19.03
                    command: ['cat']
                    tty: true
                    volumeMounts:
                    - name: docker-sock
                      mountPath: /var/run/docker.sock
                  volumes:
                  - name: docker-sock
                    hostPath:
                      path: /var/run/docker.sock
            '''
        }
    }
    stages {
        stage('Build') {
            steps {
                container('docker') {
                    sh 'docker build -t myapp:${BUILD_NUMBER} .'
                }
            }
        }
        stage('Test') {
            steps {
                sh 'npm test'
            }
        }
        stage('Deploy') {
            when { branch 'main' }
            steps {
                sh 'kubectl apply -f k8s/'
            }
        }
    }
}

Strengths:

  • Extensive plugin ecosystem
  • Highly customizable
  • On-premise deployment for security requirements
  • Jenkinsfile for pipeline-as-code

ArgoCD and GitOps

ArgoCD implements GitOps for Kubernetes deployments.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/myapp-manifests
    targetRevision: HEAD
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

ArgoCD workflow:

  1. CI pipeline builds container image and pushes to registry
  2. CI updates Kubernetes manifests in Git with new image tag
  3. ArgoCD detects Git changes and syncs to cluster
  4. ArgoCD monitors for drift and self-heals

Container-Based Workflows

Modern CI/CD embraces containers at every stage:

# Multi-stage Dockerfile for CI/CD
FROM node:14 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
RUN npm test

FROM node:14-alpine AS production
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
USER node
CMD ["node", "dist/server.js"]

Benefits:

  • Reproducible builds across environments
  • Isolated dependencies
  • Smaller production images
  • Security through minimal base images

Best Practices

Pipeline Design

  1. Keep pipelines fast: Target under 10 minutes for feedback
  2. Fail fast: Run quick checks (lint, unit tests) before slow ones
  3. Parallelize: Run independent jobs concurrently
  4. Cache dependencies: Avoid re-downloading on every build
  5. Use immutable artifacts: Build once, deploy everywhere

Security

  1. Scan dependencies: Use tools like Snyk, Dependabot, or Trivy
  2. Scan containers: Check for vulnerabilities in base images
  3. Manage secrets: Use vault solutions, never commit secrets
  4. Sign artifacts: Implement image signing for supply chain security
  5. Least privilege: CI/CD service accounts should have minimal permissions

Reliability

  1. Idempotent deployments: Running twice should produce same result
  2. Health checks: Verify deployments are healthy before proceeding
  3. Rollback automation: Automate rollback on failure detection
  4. Environment parity: Keep staging as close to production as possible
  5. Observability: Integrate logging, metrics, and tracing

References

Notes

Tool Selection Considerations

Criterion GitHub Actions GitLab CI CircleCI Jenkins
Setup Complexity Low Low Low High
Customization Medium Medium Medium High
Self-Hosting Yes Yes No Yes
Enterprise Features Yes Yes Yes Limited
Learning Curve Low Low Low Medium

Migration Path from Jenkins

Organizations moving from Jenkins typically:

  1. Start with new projects on modern platforms
  2. Gradually migrate existing pipelines
  3. Maintain Jenkins for legacy systems with custom plugins
  4. Use Jenkins as an orchestrator calling modern CI tools

Emerging Trends (Late 2020)

  • Tekton for Kubernetes-native pipelines
  • GitHub Actions gaining market share rapidly
  • Increased focus on supply chain security
  • Platform engineering and internal developer platforms
  • Ephemeral, on-demand build environments

Author: Jason Walsh

j@wal.sh

Last Updated: 2026-01-11 11:00:13

build: 2026-01-11 18:29 | sha: 48a6da1