How to Build Your First Pipeline From Scratch: A Step-by-Step CI/CD Guide

Every production issue that stems from an untested code change, a forgotten build step, or a missed lint error shares a common origin: manual execution. When steps rely on an engineer remembering to run them locally before pushing code, mistakes happen.

An automated pipeline replaces manual routine with deterministic execution. At its core, a delivery pipeline is a script triggered by repository events—such as a commit or pull request—that builds your software in a fresh environment, runs automated test suites, and flags regressions before code merges into your primary branch.

By the end of this guide, you will have a working continuous integration (CI) pipeline running from scratch using GitHub Actions and a lightweight Node.js project. You will understand how code triggers a runner, how jobs and steps execute, how to parse logs to diagnose failures, and how to verify pipeline health reliably.

Core Pipeline Concepts: How Automation Works Under the Hood

Before writing configuration files, it helps to establish a mental model of what happens when a continuous integration engine runs.

Plaintext

[Developer Push / PR] 
        │
        ▼ (Event Trigger)
┌──────────────────────────────────────────┐
│             Workflow / Pipeline          │
│                                          │
│  ┌────────────────────────────────────┐  │
│  │ Job: Build & Validate (Fresh VM)   │  │
│  │                                    │  │
│  │  1. Check out repository code      │  │
│  │  2. Install language runtime       │  │
│  │  3. Install project dependencies   │  │
│  │  4. Run code syntax / linter       │  │
│  │  5. Execute test suite             │  │
│  │  6. Package build artifact         │  │
│  └────────────────────────────────────┘  │
└──────────────────────────────────────────┘
        │
   ┌────┴────────────────────────┐
   ▼                             ▼
[Exit Code 0: PASS]          [Exit Code >0: FAIL]
Merge Allowed / Notify       Block Merge / Inspect Logs

1. Workflows, Jobs, and Steps

• Workflow (or Pipeline): The top-level automated process defined in your repository. It specifies when the automation should run (triggers) and what it should execute.
• Jobs: An isolated set of actions executed on a single host or container (known as a runner). By default, independent jobs run concurrently. If a job depends on the output of another, they are configured sequentially.
• Steps: The individual tasks inside a job. Steps execute sequentially on the same runner. If any step fails, subsequent steps are skipped by default, and the entire job terminates with a failed status.

2. Triggers (Events)

A pipeline does not run continuously in an infinite loop; it responds to declarative event hooks. Common triggers include:

• push: Whenever commits are sent to a branch.
• pull_request: When a branch comparison is opened or updated.
• workflow_dispatch: A manual trigger fired from the web interface or API.

3. Exit Codes and Pipeline State

Automated pipelines rely on Unix process exit codes:

• An exit code of 0 denotes success.
• Any non-zero exit code (e.g., 1, 127) signals a failure.

When an automated test framework or build tool encounters an error, it exits with a non-zero code. The pipeline runner intercepts this code, halts downstream execution, marks the stage with a red failure indicator, and prevents pull requests from merging.

4. CI vs. CD

• Continuous Integration (CI): Focuses on validating changes. It checks out code, installs dependencies, validates syntax, compiles binaries (if applicable), and runs tests. Its objective is early regression detection.
• Continuous Delivery / Deployment (CD): Takes the validated output (an artifact, binary, or container image) and releases it to staging or production environments.

A reliable CI pipeline is a mandatory prerequisite for CD. In this tutorial, we build a foundational CI pipeline.

Prerequisites and Tech Stack Selection

To keep the pipeline fast, zero-cost, and easy to run without provisioning external servers, we will use:

1. GitHub Actions: It requires no self-hosted infrastructure. Free-tier accounts include generous monthly runner minutes, and workflow definitions live directly inside the repository as YAML files.
2. Node.js (LTS): JavaScript/TypeScript has zero-dependency runtime requirements, clear package management via npm, and readable testing libraries.
3. Git and a GitHub Account: You need local Git installed and an authenticated GitHub account to push repositories.

No cloud provider accounts (AWS/GCP/Azure), external hosting platforms, or credit cards are required.

Step 1: Initialize the Application Project

A pipeline needs code to evaluate. We will create a minimal Node.js utility containing a functional module and an automated test suite.

1. Initialize the Directory and Manifest

Create an empty directory on your local machine and initialize an npm package:

Bash

mkdir first-pipeline-project
cd first-pipeline-project
npm init -y

Open package.json and replace its contents with the following:

JSON

{
  "name": "first-pipeline-project",
  "version": "1.0.0",
  "description": "Demonstration project for learning CI/CD pipelines",
  "main": "src/calculator.js",
  "scripts": {
    "test": "node --test test/calculator.test.js",
    "lint": "node -c src/calculator.js test/calculator.test.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

Notice the two scripts:

• lint: Uses the Node.js compiler check (node -c) to validate JavaScript syntax without executing the files.
• test: Uses Node’s native, zero-dependency test runner (node --test), introduced in Node 18+.

2. Add the Application Logic

Create a src/ directory and add calculator.js:

Bash

mkdir src test

Paste the following logic into src/calculator.js:

JavaScript

/**
 * Simple calculation utility functions.
 */
function add(a, b) {
  if (typeof a !== 'number' || typeof b !== 'number') {
    throw new TypeError('Both arguments must be numbers');
  }
  return a + b;
}

function subtract(a, b) {
  if (typeof a !== 'number' || typeof b !== 'number') {
    throw new TypeError('Both arguments must be numbers');
  }
  return a - b;
}

module.exports = { add, subtract };

3. Add the Automated Test Suite

Open test/calculator.test.js and add unit tests that verify both happy paths and error handling:

JavaScript

const test = require('node:test');
const assert = require('node:assert/strict');
const { add, subtract } = require('../src/calculator');

test('add() correctly sums two numbers', () => {
  assert.equal(add(2, 3), 5);
  assert.equal(add(-1, 1), 0);
});

test('add() throws an error when given non-numeric inputs', () => {
  assert.throws(() => add('2', 3), {
    name: 'TypeError',
    message: 'Both arguments must be numbers',
  });
});

test('subtract() correctly deducts second number from first', () => {
  assert.equal(subtract(10, 4), 6);
  assert.equal(subtract(0, 5), -5);
});

4. Verify Local Execution

Always verify that your validation commands function on your local workstation before automating them in a pipeline. Run:

Bash

npm run lint
npm test

Expected output:

Plaintext

✔ add() correctly sums two numbers (0.8ms)
✔ add() throws an error when given non-numeric inputs (0.2ms)
✔ subtract() correctly deducts second number from first (0.2ms)
ℹ tests 3
ℹ suites 0
ℹ pass 3
ℹ fail 0

Step 2: Establish Source Control

Pipelines exist in context with version control. Initialize your local Git repository and create a .gitignore file to ensure untracked build artifacts and dependencies are not committed.

Create .gitignore:

Plaintext

node_modules/
.DS_Store
*.log

Commit the codebase locally:

Bash

git init
git add .
git commit -m "feat: initial project structure with calculator module and tests"

Next, create a new public or private repository on GitHub named first-pipeline-project, connect your remote origin, and push:

Bash

git branch -M main
git remote add origin https://github.com/<YOUR-USERNAME>/first-pipeline-project.git
git push -u origin main

Step 3: Write the Pipeline Configuration File

GitHub Actions looks for workflow definitions in a dedicated directory: .github/workflows/. Configuration files must use YAML syntax (.yml or .yaml).

1. Create the Directory and File

Bash

mkdir -p .github/workflows
touch .github/workflows/ci.yml

2. Add the Pipeline Specification

Add the following configuration to .github/workflows/ci.yml:

YAML

name: Continuous Integration

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

permissions:
  contents: read

jobs:
  validate-and-test:
    runs-on: ubuntu-latest

    steps:
      - name: Check out repository code
        uses: actions/checkout@v4

      - name: Set up Node.js runtime
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Validate code syntax (Lint)
        run: npm run lint

      - name: Execute automated unit tests
        run: npm test

Step 4: Deconstruct the Configuration File

To manage and debug pipelines effectively, you need to understand each line of the configuration:

Section / Key	Purpose	Technical Explanation
name	Human-readable pipeline label	Displayed in GitHub’s “Actions” tab UI.
on	Event triggers	Specifies that this workflow runs whenever commits are pushed directly to main or when a Pull Request targets main.
permissions	Security boundary	Enforces the Principle of Least Privilege. Setting contents: read prevents the pipeline runner from modifying code or writing tags unexpectedly.
jobs	Execution containers	Defines one or more discrete jobs. Here, we declare a single job named validate-and-test.
runs-on: ubuntu-latest	Virtual machine host	Instructs GitHub to provision an ephemeral, cleanly imaged Ubuntu Linux runner for this job execution.
actions/checkout@v4	Source checkout step	Clones your Git repository onto the runner’s filesystem so downstream tasks can access files.
actions/setup-node@v4	Language runtime	Downloads and configures the Node.js v20 engine into the runner’s $PATH. The cache: 'npm' parameter speeds up execution by caching package archives across runs.
run: npm ci	Dependency installation	Uses clean install (npm ci) rather than npm install. This checks your package-lock.json strictly to ensure the runner matches your exact local dependency tree.
run: npm run lint	Static validation	Executes our syntax validation script. If a file has a syntax error, this step exits with code 1, halting the pipeline immediately.
run: npm test	Dynamic validation	Runs the unit tests. If any assertion fails, the test runner exits with code 1, immediately marking the build as failed.

Step 5: Execute and Validate the Pipeline

With the workflow file saved, commit and push it to trigger execution.

Bash

git add .github/workflows/ci.yml
git commit -m "ci: add GitHub Actions workflow configuration"
git push origin main

Inspecting the Run

1. Navigate to your repository on GitHub in your web browser.
2. Click on the Actions tab along the top navigation bar.
3. You will see an active or recently completed workflow labeled Continuous Integration.
4. Click on the workflow run, then click on the validate-and-test job in the left-hand panel.

Plaintext

✓ Set up job
✓ Check out repository code
✓ Set up Node.js runtime
✓ Install dependencies
✓ Validate code syntax (Lint)
✓ Execute automated unit tests
✓ Complete job

Each step can be expanded to view standard output (stdout) and standard error (stderr) generated directly by the underlying Linux runner.

Step 6: Test Failure Handling (Simulating a Broken Build)

A pipeline that only passes tells you that your code works today. You must also prove that it successfully catches broken code before trusting it with your software delivery.

1. Introduce an Intentional Bug

Create a new feature branch:

Bash

git checkout -b break-the-calculator

Open src/calculator.js and modify the add function so that it introduces a logic bug:

JavaScript

function add(a, b) {
  if (typeof a !== 'number' || typeof b !== 'number') {
    throw new TypeError('Both arguments must be numbers');
  }
  return a - b; // BUG: Changed addition to subtraction
}

2. Push and Open a Pull Request

Commit the regression and push the branch:

Bash

git add src/calculator.js
git commit -m "refactor: optimize arithmetic operations"
git push origin break-the-calculator

Navigate to GitHub and open a Pull Request comparing break-the-calculator into main.

3. Observe the Failed Pipeline

Within seconds, the pull_request event triggers the workflow. Because of our intentional logic bug, the assertion inside test/calculator.test.js fails:

Plaintext

✖ add() correctly sums two numbers (1.2ms)
  AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
  -1 !== 5
      at TestContext.<anonymous> (test/calculator.test.js:6:10)

ℹ tests 3
ℹ suites 0
ℹ pass 2
ℹ fail 1
Error: Process completed with exit code 1.

GitHub automatically flags the pull request with a red cross, alerting reviewers that the change contains an unhandled regression and should not be merged.

Practical Troubleshooting Guide

When building pipelines from scratch, things rarely work on the first try. Use this reference matrix to quickly diagnose common failures:

Symptom	Probable Cause	Corrective Action
Workflow does not trigger on push	Incorrect directory structure or filename	Ensure the path is exactly .github/workflows/<name>.yml. Verify that hidden folders (.github) are tracked by Git.
Invalid YAML syntax error	Indentation or tab characters	YAML strictly forbids tab characters (\t). Use 2 spaces for indentation. Validate formatting using an online linter or CLI tool like yamllint.
Command not found (Exit code 127)	Missing runtime setup step	If running node, npm, python, or go, confirm that the corresponding setup action (e.g., actions/setup-node) executed before calling the tool.
npm ci fails on the runner	Missing package-lock.json	npm ci requires an existing package-lock.json file in version control. Run npm install locally, commit the lockfile, and push.
File or path not found	Case-sensitivity mismatch	Ubuntu runners use case-sensitive filesystems. If your local machine runs macOS or Windows (case-insensitive), require('./Calculator') will work locally but crash in CI. Align all file casing.
Permission Denied (HTTP 403)	Insufficient default runner permissions	Check the permissions: block in your workflow. For basic CI that only reads source code, contents: read is sufficient.

Essential Security and Maintenance Best Practices

Once your first pipeline is running smoothly, apply these fundamental engineering practices to keep it secure and maintainable:

1. Pin Actions to Explicit Major Versions or SHAs

Avoid unversioned actions or floating @latest tags (e.g., actions/checkout@latest). If an upstream dependency is compromised or introduces a breaking update, your pipeline will fail unexpectedly. Always pin actions to a specific major version tag (@v4) or an immutable commit SHA:

YAML

uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1

2. Never Hardcode Sensitive Credentials

Never place API keys, private certificates, or database passwords in source code or YAML configuration files. Store them under Settings > Secrets and variables > Actions in your repository, and inject them into specific steps using environment variables:

YAML

- name: Run Integration Suite
  env:
    API_KEY: ${{ secrets.THIRD_PARTY_API_KEY }}
  run: npm run test:integration

3. Keep Workflow Files Small and Purpose-Driven

Avoid inlining long shell scripts directly inside YAML run steps. If a build task requires more than 5–10 lines of procedural shell commands, extract it into an executable script file (e.g., ./scripts/build.sh), test that script locally, and call the script from your pipeline step.

Next Steps for Maturing Your Delivery Pipeline

A working build-and-test CI check provides a solid automation foundation. As your application evolves, consider adding the following stages in logical progression:

Plaintext

Phase 1 (Current)        Phase 2 (Quality & Security)      Phase 3 (Delivery)
┌─────────────────┐      ┌───────────────────────────┐     ┌───────────────────────┐
│ • Checkout Code │ ───► │ • Static Analysis (ESLint)│ ──► │ • Build Container     │
│ • Unit Tests    │      │ • Dependency Audit (npm)  │     │ • Push to Registry    │
│ • Syntax Check  │      │ • Code Coverage Reports   │     │ • Deploy to Staging   │
└─────────────────┘      └───────────────────────────┘     └───────────────────────┘

1. Static Analysis & Linting Tools: Incorporate mature linters (such as ESLint or Prettier) to enforce consistent code formatting and flag anti-patterns across the team.
2. Automated Vulnerability Scanning: Add an npm audit step or integrate tools like Trivy or Snyk to alert you when third-party packages contain published CVEs.
3. Artifact Archiving: Use actions/upload-artifact@v4 to store compiled distributions, coverage reports, or logs so they can be downloaded and reviewed after the run finishes.
4. Environment Deployment: Add a downstream deployment job that triggers only when commits merge into your production branch, deploying your code automatically to staging or production environments.

Frequently Asked Questions

Which tool is best for building a first pipeline?

GitHub Actions and GitLab CI/CD are the best starting points for beginners because their configuration files live directly inside your repository, and they provide managed runner infrastructure out of the box. Traditional on-premise engines like Jenkins require significant server administration, network configuration, and security patching before you can run your first job.

Can I build a CI pipeline without deploying to a live server?

Yes. In fact, you should start without deployment. A continuous integration pipeline’s primary goal is verification: confirming that changes compile, adhere to coding standards, and pass automated tests. Deployment is a secondary phase (Continuous Delivery) that should only occur after your CI checks pass consistently.

Why did my pipeline pass locally but fail in CI?

This discrepancy usually comes down to environment drift:

1. Uncommitted files: You have local files or configs that are listed in .gitignore or were never added to Git.
2. Global dependencies: You have software installed globally on your workstation that the fresh CI runner does not have.
3. OS-level differences: Your workstation OS (macOS/Windows) handles path separators and filename casing differently than the Linux host running inside CI.

How do automated tests stop broken code from reaching production?

Pipeline stages run sequentially. When an assertion in an automated test fails, the test process exits with an exit code greater than 0. The pipeline runner registers this status, immediately terminates downstream steps (such as deployment or packaging), marks the run as failed, and reports this failure to your branch protection rules to block merging.

Summary

Building an automated pipeline does not require complex cloud architecture or enterprise toolchains. By defining clear triggers, provisioning a managed runner, and codifying your build, lint, and test steps into a declarative configuration file, you establish a deterministic safety net for your project. The pipeline you built today checks out code, sets up runtimes, validates syntax, and prevents broken logic from merging into your codebase. Treat your pipeline configuration as first-class code: keep it versioned, test it periodically by introducing intentional regressions, and gradually expand its capabilities as your system grows.