CircleCI CI/CD Tutorial

CircleCI is a modern continuous integration and continuous deployment (CI/CD) platform that automates the software development process. It enables teams to build, test, and deploy code quickly and reliably through automated pipelines defined in version control.

Why CircleCI?

CircleCI offers cloud-native builds, extensive Docker support, parallel job execution, and seamless integration with popular version control systems like GitHub and Bitbucket.

1. What is CI/CD?

Continuous Integration (CI) and Continuous Deployment (CD) are software development practices that help teams deliver code changes more frequently and reliably.

Practice	Description	Benefits
Continuous Integration	Automatically build and test code changes	Early bug detection, reduced integration issues
Continuous Deployment	Automatically deploy passing builds to production	Faster releases, reduced manual errors
Continuous Delivery	Automatically prepare code for deployment	Deployment-ready code at any time

CI/CD Pipeline Stages

Code Push → Build → Test → Security Scan → Deploy to Staging → Deploy to Production

2. CircleCI Architecture

GitHub/Bitbucket → CircleCI → Docker Containers/VMs → Deploy to Cloud
                        │
                        ├─ Build Jobs
                        ├─ Test Jobs  
                        ├─ Deploy Jobs
                        └─ Workflows (Orchestration)

Key Components:

Orbs: Reusable configuration packages
Jobs: Individual units of work (build, test, deploy)
Steps: Commands executed within a job
Workflows: Orchestration of multiple jobs
Executors: Environment where jobs run (Docker, Machine, macOS)

3. Getting Started

3.1 Prerequisites

GitHub or Bitbucket account
Project repository
CircleCI account (sign up at circleci.com)

3.2 Initial Setup

Sign up for CircleCI
- Visit https://circleci.com/signup
- Connect your GitHub or Bitbucket account
- Authorize CircleCI to access your repositories
Set up a project
- Select your repository from the dashboard
- Click "Set Up Project"
- Choose a configuration template or start from scratch
Create configuration file
- Add .circleci/config.yml to your repository root
- Commit and push to trigger your first build

4. Basic Configuration

4.1 Minimal Configuration

.circleci/config.yml
version: 2.1

jobs:
  build:
    docker:
      - image: cimg/node:18.0
    steps:
      - checkout
      - run:
          name: Install dependencies
          command: npm install
      - run:
          name: Run tests
          command: npm test

workflows:
  build-and-test:
    jobs:
      - build

4.2 Configuration Structure Explained

Section	Purpose	Required
`version`	Config version (use 2.1 for latest features)	Yes
`jobs`	Define individual jobs	Yes
`workflows`	Orchestrate job execution	No (single job)
`orbs`	Import reusable packages	No
`executors`	Define reusable execution environments	No

5. Working with Docker

5.1 Using Docker Images

Node.js
Python
Go

jobs:
  build:
    docker:
      - image: cimg/node:18.0
    steps:
      - checkout
      - run: npm ci
      - run: npm test

jobs:
  build:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run: pip install -r requirements.txt
      - run: pytest

jobs:
  build:
    docker:
      - image: cimg/go:1.21
    steps:
      - checkout
      - run: go mod download
      - run: go test ./...

5.2 Multiple Containers (Services)

jobs:
  test:
    docker:
      - image: cimg/python:3.11
      - image: cimg/postgres:14.0
        environment:
          POSTGRES_USER: testuser
          POSTGRES_PASSWORD: testpass
          POSTGRES_DB: testdb
    steps:
      - checkout
      - run:
          name: Wait for Postgres
          command: |
            dockerize -wait tcp://localhost:5432 -timeout 1m
      - run:
          name: Run integration tests
          command: pytest tests/integration

6. Advanced Job Configuration

6.1 Caching Dependencies

jobs:
  build:
    docker:
      - image: cimg/node:18.0
    steps:
      - checkout
      
      # Restore cached dependencies
      - restore_cache:
          keys:
            - v1-dependencies-{{ checksum "package-lock.json" }}
            - v1-dependencies-
      
      - run: npm ci
      
      # Save dependencies to cache
      - save_cache:
          paths:
            - node_modules
          key: v1-dependencies-{{ checksum "package-lock.json" }}
      
      - run: npm test

6.2 Parallel Job Execution

jobs:
  test:
    parallelism: 4
    docker:
      - image: cimg/node:18.0
    steps:
      - checkout
      - run: npm ci
      - run:
          name: Run tests in parallel
          command: |
            TESTFILES=$(circleci tests glob "test/**/*.test.js" | circleci tests split --split-by=timings)
            npm test $TESTFILES

6.3 Storing Artifacts

jobs:
  build:
    docker:
      - image: cimg/node:18.0
    steps:
      - checkout
      - run: npm ci
      - run: npm run build
      
      # Store build artifacts
      - store_artifacts:
          path: dist
          destination: build-output
      
      # Store test results
      - store_test_results:
          path: test-results

7. Workflows and Orchestration

7.1 Sequential Workflow

workflows:
  version: 2
  build-test-deploy:
    jobs:
      - build
      - test:
          requires:
            - build
      - deploy:
          requires:
            - test
          filters:
            branches:
              only: main

7.2 Parallel Workflow with Fan-out/Fan-in

workflows:
  version: 2
  parallel-testing:
    jobs:
      - build
      - test-unit:
          requires:
            - build
      - test-integration:
          requires:
            - build
      - test-e2e:
          requires:
            - build
      - deploy:
          requires:
            - test-unit
            - test-integration
            - test-e2e
          filters:
            branches:
              only: main

7.3 Scheduled Workflows (Cron)

workflows:
  nightly-build:
    triggers:
      - schedule:
          cron: "0 0 * * *"
          filters:
            branches:
              only:
                - main
    jobs:
      - build
      - test

8. Environment Variables and Secrets

8.1 Project Environment Variables

Set in CircleCI UI:

Navigate to Project Settings → Environment Variables
Add variables (e.g., API_KEY, DATABASE_URL)
Reference in config:

jobs:
  deploy:
    docker:
      - image: cimg/base:stable
    steps:
      - run:
          name: Deploy to production
          command: |
            echo "Deploying with API_KEY: $API_KEY"
            ./deploy.sh

8.2 Context-based Secrets

workflows:
  deploy-production:
    jobs:
      - deploy:
          context: production-secrets
          filters:
            branches:
              only: main

8.3 Inline Environment Variables

jobs:
  build:
    docker:
      - image: cimg/node:18.0
    environment:
      NODE_ENV: production
      LOG_LEVEL: info
    steps:
      - checkout
      - run: npm ci
      - run: npm run build

9. Using Orbs

Orbs are reusable packages of CircleCI configuration that simplify common tasks.

9.1 Popular Orbs

version: 2.1

orbs:
  node: circleci/node@5.1.0
  aws-cli: circleci/aws-cli@3.1.0
  slack: circleci/slack@4.12.0

jobs:
  build-and-deploy:
    executor: node/default
    steps:
      - checkout
      - node/install-packages
      - run: npm run build
      - aws-cli/setup
      - run: aws s3 sync ./dist s3://my-bucket
      - slack/notify:
          event: pass
          template: basic_success_1

9.2 Custom Orbs

You can create organization-specific orbs for sharing configuration across projects.

10. Real-World Examples

10.1 Node.js Web Application

version: 2.1

orbs:
  node: circleci/node@5.1.0
  aws-s3: circleci/aws-s3@3.1.0

jobs:
  build:
    executor: node/default
    steps:
      - checkout
      - node/install-packages:
          pkg-manager: npm
      - run:
          name: Build application
          command: npm run build
      - persist_to_workspace:
          root: .
          paths:
            - dist

  test:
    executor: node/default
    steps:
      - checkout
      - node/install-packages
      - run:
          name: Run unit tests
          command: npm run test:unit
      - run:
          name: Run integration tests
          command: npm run test:integration
      - store_test_results:
          path: test-results

  deploy:
    executor: aws-s3/default
    steps:
      - attach_workspace:
          at: .
      - aws-s3/sync:
          from: dist
          to: s3://my-app-bucket
          arguments: --acl public-read --cache-control "max-age=86400"

workflows:
  build-test-deploy:
    jobs:
      - build
      - test:
          requires:
            - build
      - deploy:
          requires:
            - test
          filters:
            branches:
              only: main

10.2 Python Machine Learning Pipeline

version: 2.1

jobs:
  train:
    docker:
      - image: cimg/python:3.11
    resource_class: large
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-pip-{{ checksum "requirements.txt" }}
      - run:
          name: Install dependencies
          command: |
            python -m venv venv
            . venv/bin/activate
            pip install -r requirements.txt
      - save_cache:
          paths:
            - venv
          key: v1-pip-{{ checksum "requirements.txt" }}
      - run:
          name: Train model
          command: |
            . venv/bin/activate
            python train.py
      - store_artifacts:
          path: models
          destination: trained-models
      - persist_to_workspace:
          root: .
          paths:
            - models

  evaluate:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - attach_workspace:
          at: .
      - restore_cache:
          keys:
            - v1-pip-{{ checksum "requirements.txt" }}
      - run:
          name: Evaluate model
          command: |
            . venv/bin/activate
            python evaluate.py
      - store_test_results:
          path: evaluation-results

  deploy:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run:
          name: Deploy model
          command: |
            # Deploy to MLflow, SageMaker, or other platform
            ./deploy_model.sh

workflows:
  ml-pipeline:
    jobs:
      - train
      - evaluate:
          requires:
            - train
      - deploy:
          requires:
            - evaluate
          filters:
            branches:
              only: main

10.3 Docker Image Build and Push

version: 2.1

orbs:
  docker: circleci/docker@2.2.0

jobs:
  build-and-push:
    executor: docker/docker
    steps:
      - checkout
      - setup_remote_docker:
          version: 20.10.14
      - docker/check
      - docker/build:
          image: myapp
          tag: << pipeline.git.revision >>
      - docker/push:
          image: myapp
          tag: << pipeline.git.revision >>

workflows:
  build-image:
    jobs:
      - build-and-push:
          filters:
            branches:
              only: main

11. Best Practices

11.1 Configuration

Practice	Why	Example
Use caching	Speed up builds	Cache dependencies, build outputs
Pin image versions	Reproducible builds	`cimg/node:18.0` not `cimg/node:latest`
Keep jobs small	Better parallelization	Separate build, test, deploy
Use workspaces	Share data between jobs	Build once, deploy many
Leverage orbs	DRY principle	Reuse common configurations

11.2 Security

# ❌ Bad: Hardcoded secrets
jobs:
  deploy:
    steps:
      - run: echo "API_KEY=abc123" > .env

# ✅ Good: Environment variables
jobs:
  deploy:
    steps:
      - run: echo "API_KEY=$API_KEY" > .env

11.3 Resource Optimization

# Use appropriate resource classes
jobs:
  light-task:
    resource_class: small
    steps:
      - run: npm test

  heavy-task:
    resource_class: large
    steps:
      - run: ./compile-project.sh

12. Monitoring and Debugging

12.1 SSH Debugging

Enable SSH access to failed builds:

jobs:
  debug:
    steps:
      - checkout
      - run: npm test
      # In CircleCI UI, click "Rerun job with SSH"

12.2 Insights and Analytics

CircleCI provides built-in insights:

Duration Trends: Track build time over time
Success Rate: Monitor pipeline health
Credit Usage: Optimize resource consumption
Flaky Tests: Identify unstable tests

12.3 Notifications

version: 2.1

orbs:
  slack: circleci/slack@4.12.0

jobs:
  test:
    steps:
      - checkout
      - run: npm test
      - slack/notify:
          event: fail
          custom: |
            {
              "text": "Build failed on branch: $CIRCLE_BRANCH"
            }

13. Migration from Other CI/CD Tools

13.1 From Jenkins

Jenkins	CircleCI
Jenkinsfile	.circleci/config.yml
stage	job
steps	steps
agent	executor
pipeline	workflow

13.2 From GitHub Actions

GitHub Actions	CircleCI
.github/workflows	.circleci/config.yml
jobs	jobs
steps	steps
runs-on	executor
needs	requires

14. Common Issues and Solutions

Issue	Cause	Solution
Slow builds	No caching	Implement dependency caching
Flaky tests	Race conditions	Use proper test isolation
Out of credits	Resource intensive jobs	Optimize parallelism, use smaller executors
Failed deployments	Missing environment variables	Verify secrets in project settings
Docker build failures	Layer caching issues	Use setup_remote_docker correctly

15. Advanced Features

15.1 Dynamic Configuration

version: 2.1

setup: true

orbs:
  continuation: circleci/continuation@0.3.1

jobs:
  setup:
    executor: continuation/default
    steps:
      - checkout
      - run:
          name: Generate config
          command: |
            # Generate dynamic configuration
            ./generate-config.sh > generated-config.yml
      - continuation/continue:
          configuration_path: generated-config.yml

15.2 Matrix Jobs

version: 2.1

workflows:
  test-matrix:
    jobs:
      - test:
          matrix:
            parameters:
              node-version: ["16.0", "18.0", "20.0"]
              os: ["linux", "macos"]

15.3 Private Orbs for Organizations

Create reusable configurations specific to your organization:

# In your private orb repository
version: 2.1

description: Company standard deployment orb

commands:
  deploy:
    parameters:
      environment:
        type: string
    steps:
      - run: ./deploy.sh << parameters.environment >>

16. Cost Optimization

16.1 Credit Management

Use smaller resource classes when possible
Implement effective caching strategies
Run expensive jobs only on main branch
Leverage parallelism efficiently
Monitor credit usage in CircleCI dashboard

16.2 Example: Branch-based Resource Allocation

jobs:
  test:
    docker:
      - image: cimg/node:18.0
    resource_class: << pipeline.git.branch == "main" ? "large" : "medium" >>

17. Integration Examples

17.1 Kubernetes Deployment

jobs:
  deploy-k8s:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Install kubectl
          command: |
            curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
            chmod +x kubectl
            sudo mv kubectl /usr/local/bin/
      - run:
          name: Deploy to Kubernetes
          command: |
            echo $KUBE_CONFIG | base64 -d > kubeconfig
            kubectl --kubeconfig=kubeconfig apply -f k8s/

17.2 Terraform Infrastructure

jobs:
  terraform-apply:
    docker:
      - image: hashicorp/terraform:1.5
    steps:
      - checkout
      - run:
          name: Terraform Init
          command: terraform init
      - run:
          name: Terraform Plan
          command: terraform plan -out=tfplan
      - run:
          name: Terraform Apply
          command: terraform apply tfplan

18. Frequently Asked Questions

Q: How do I access build artifacts from other jobs?
A: Use persist_to_workspace in the producing job and attach_workspace in the consuming job.

Q: Can I run CircleCI locally?
A: Yes, use the CircleCI CLI with circleci local execute for basic testing.

Q: How do I handle monorepos?
A: Use path filtering orb or dynamic configuration to trigger jobs based on changed files.

Q: What's the difference between workspace and cache?
A: Workspace shares data between jobs in the same workflow; cache persists across workflow runs.

19. Additional Resources

20. Next Steps

Set up your first CircleCI pipeline
Explore advanced orbs for your tech stack
Implement parallel testing for faster builds
Configure deployment to your preferred platform
Monitor and optimize pipeline performance

This guide reflects CircleCI best practices as of 2025. Always verify configurations against the latest CircleCI documentation for your specific use case.

Getting Help

Join the CircleCI Community Forum or check the Support Center for additional assistance.

1. What is CI/CD?​

CI/CD Pipeline Stages​

2. CircleCI Architecture​

3. Getting Started​

3.1 Prerequisites​

3.2 Initial Setup​

4. Basic Configuration​

4.1 Minimal Configuration​

4.2 Configuration Structure Explained​

5. Working with Docker​

5.1 Using Docker Images​

5.2 Multiple Containers (Services)​

6. Advanced Job Configuration​

6.1 Caching Dependencies​

6.2 Parallel Job Execution​

6.3 Storing Artifacts​

7. Workflows and Orchestration​

7.1 Sequential Workflow​

7.2 Parallel Workflow with Fan-out/Fan-in​

7.3 Scheduled Workflows (Cron)​

8. Environment Variables and Secrets​

8.1 Project Environment Variables​

8.2 Context-based Secrets​

8.3 Inline Environment Variables​

9. Using Orbs​

9.1 Popular Orbs​

9.2 Custom Orbs​

10. Real-World Examples​

10.1 Node.js Web Application​

10.2 Python Machine Learning Pipeline​

10.3 Docker Image Build and Push​

11. Best Practices​

11.1 Configuration​

11.2 Security​

11.3 Resource Optimization​

12. Monitoring and Debugging​

12.1 SSH Debugging​

12.2 Insights and Analytics​

12.3 Notifications​

13. Migration from Other CI/CD Tools​

13.1 From Jenkins​

13.2 From GitHub Actions​

14. Common Issues and Solutions​

15. Advanced Features​

15.1 Dynamic Configuration​

15.2 Matrix Jobs​

15.3 Private Orbs for Organizations​

16. Cost Optimization​

16.1 Credit Management​

16.2 Example: Branch-based Resource Allocation​

17. Integration Examples​

17.1 Kubernetes Deployment​

17.2 Terraform Infrastructure​

18. Frequently Asked Questions​

19. Additional Resources​

20. Next Steps​