Skip to main content
Navigation
HomeTechnical ReferenceJournalGitHubGitHub
Sidebar — toggle document categories via the logo
Multi-Strategy Deploy Pipeline

Multi-Strategy Deploy Pipeline

March 8, 2025

Architecture

The pipeline lives in references/scripts/ under the repo. It uses a module-loader patternlib/index.sh sources every .sh module in sorted order, guarded against double-includes. The entry point (deploy.sh) sources the loader, then calls into whichever modules it needs.

references/scripts/
├── deploy.sh # Entry point / orchestrator
└── lib/
├── index.sh # Auto-loader (sources all *.sh in lib/)
├── helpers.sh # die(), dry-run, pushd/popd, timestamps
├── docker.sh # Build & push strategy per environment
├── kube.sh # kubectl apply (local or remote via SSH)
├── systemd.sh # SSH + systemd image-bump for bare-metal
├── version.sh # .env / deploy.yaml / .version snapshot
├── git.sh # Git add + commit
└── help.sh # Usage text

The loader pattern means you never need to manually source individual modules — adding a new lib/*.sh file automatically makes its functions available.

How the loader works (lib/index.sh)

_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
while IFS= read -r -d '' mod; do
case "$(basename "$mod")" in index.sh) continue ;; esac
. "$mod"
done < <(find "$_LIB_DIR" -maxdepth 1 -type f -name '*.sh' -print0 | sort -z)

It finds its own directory (handles being sourced from anywhere), then globs every .sh file in the same folder — except itself — and sources them in deterministic alphabetical order. A __LIB_INDEX_SH_SOURCED guard variable prevents double-loading if index.sh is sourced twice in the same shell.


Environments & Deployment Paths

The pipeline supports three environments, each with a different rollout strategy:

EnvironmentDockerfileVersion bumpRollout strategy
localDockerfile.local.env + kubernetes/deploy.yaml + .versionkubectl apply on local cluster
stagingDockerfile.staging.env + kubernetes/deploy.yaml + .versionSCP manifests to remote → kubectl apply via SSH
productionDockerfile.prod.env only + .versionSSH → edit /etc/default/<svc>systemctl restart

Staging uses Kubernetes. Manifests are SCP'd to a remote control node and applied with kubectl apply -k (Kustomize).

Production uses bare-metal VMs. The pipeline SSHs directly into each host, atomically bumps the IMAGE= line in /etc/default/<service>, then runs systemctl daemon-reload + restart.

Each service directory must contain the corresponding Dockerfile.* or the build fails immediately. You define which directories constitute "services" inside build_all_images().


Quick Reference — All Commands

# Basic usage pattern
./deploy.sh <version> <env> [options]

# Local development cluster
./deploy.sh 1.4.0 local

# Staging (Kubernetes via SSH)
./deploy.sh 1.4.0 staging

# Production (bare-metal via systemd)
./deploy.sh 1.4.0 production

# Dry-run: print what would happen, make no changes
./deploy.sh --dry-run 1.4.0 staging
DRY_RUN=1 ./deploy.sh 1.4.0 local

# Show help
./deploy.sh --help

Positional arguments:

  • version — Semver tag used as the Docker image tag (e.g. 1.4.0)
  • env — One of local, staging, production

Options:

  • -n, --dry-run — Echo commands instead of executing them
  • -h, --help, help — Print usage and exit

Environment variable:

  • DRY_RUN=1 — Same effect as --dry-run

Deploy to Kubernetes (local / staging)

Both local and staging follow the same pattern, differing only in whether kubectl runs on the local machine or over SSH.

Flow

build_all_images → push_all_images (staging only)


update_envs → bumps version in .env files
→ bumps image tags in deploy.yaml
→ writes .version snapshot


exec_kube → copies ./kubernetes/* to the target
→ kubectl apply -k (Kustomize overlay)

The Kubernetes module (lib/kube.sh)

exec_kube() {
local remote_host="$1"
[ -z "$remote_host" ] && die "Remote host not specified."

if [ "$remote_host" = "local" ]; then
maybe rm -rf ~/kubernetes/<project>/*
maybe cp -r ./kubernetes/* ~/kubernetes/<project>/
maybe kubectl apply -k ~/kubernetes/<project> --force
else
maybe ssh "$remote_host" rm -rf ~/kubernetes/<project>/*
maybe scp -r -v ./kubernetes/* "$remote_host":~/kubernetes/<project>/
maybe ssh "$remote_host" 'kubectl apply -k ~/kubernetes/<project> --force'
fi
}

Key points:

  • Uses Kustomize (-k) — the ./kubernetes/ directory should contain a kustomization.yaml
  • For local: copies manifests to the target path and applies directly
  • For remote: SCPs manifests, then SSHs and applies
  • --force flag ensures resource conflicts are resolved
  • The target path (~/kubernetes/<project>/) is a project-specific value — change it to match your layout

Example: staging deploy

./deploy.sh 1.5.0 staging

What happens:

  1. Builds each service with Dockerfile.staging
  2. Pushes images to your registry
  3. Updates version strings in all .env files
  4. Updates image tags in kubernetes/deploy.yaml
  5. Writes .version metadata file
  6. SCPs manifests to the staging control host
  7. Runs kubectl apply -k on the staging cluster
  8. Git commits everything

Deploy to SSH Hosts (production)

Production bypasses Kubernetes entirely — it uses SSH + systemd to update bare-metal VMs running services directly.

Flow

build_all_images → push_all_images


update_envs → bumps .env files (skips kubernetes/deploy.yaml)


systemd_update_remote_host → SSHs to each production host
→ edits IMAGE= in /etc/default/<service>
→ systemctl daemon-reload
→ systemctl restart <service>


git_commit

The systemd module (lib/systemd.sh)

For each service listed in systemd_update_remote_host() on each production host:

  1. SSH into the host with BatchMode=yes (key-based auth)
  2. Check /etc/default/<service> exists
  3. Backup the env file: sudo cp -a "$env_file" "${env_file}.bak"
  4. Find the first IMAGE= line
  5. Replace the tag portion (everything after :) with the new version
  6. Write the update via sudo tee + atomic mv
  7. Systemctl sequence:
    sudo systemctl daemon-reload
    sudo systemctl restart "${svc}.service"
    sudo systemctl --no-pager --full status "${svc}.service"

The full update logic runs remotely via a heredoc — no scripts need to be pre-installed on the target host:

maybye ssh "${SSH_OPTS[@]}" "$host" bash -s -- "$svc" "$new_tag" <<'REMOTE'
set -euo pipefail
svc="$1"
new_tag="$2"
# ... bump IMAGE=, daemon-reload, restart ...
REMOTE

SSH options used:

  • BatchMode=yes — never prompt for password (fails if key auth isn't set up)
  • StrictHostKeyChecking=accept-new — auto-accept first connection, fail on host-key mismatch
  • ConnectTimeout=8 — fail fast if unreachable

Customizing hosts and services

The hostnames and service names are hard-coded near the top of deploy.sh and inside systemd.sh. Adapt these for your own project:

# deploy.sh — remote hosts
STAGING_HOST="your-k8s-control-node"
PRODUCTION_HOST_1="your-vm-1"
PRODUCTION_HOST_2="your-vm-2"

# systemd.sh — service names (must match /etc/default/<name>)
for svc in your-api your-frontend your-worker; do
_systemd_remote_update_one "$host" "$svc" "$new_tag"
done

Example: production deploy

./deploy.sh 2.0.0 production

What happens:

  1. Builds all images with Dockerfile.prod
  2. Pushes to registry
  3. Updates .env files (skips kubernetes/deploy.yaml since UPDATE_KUBE=false)
  4. Writes .version
  5. For each production host:
    • SSHs in, bumps IMAGE= in /etc/default/<service>
    • Restarts the service via systemctl
    • Repeats for every service
  6. Git commits

The Dry-Run System

Every destructive or state-changing operation is wrapped in maybe:

maybe() {
if is_dry_run; then
echo "[DRY-RUN] $*"
else
"$@"
fi
}

This means --dry-run gives you a complete trace of every command that would execute — including docker build, docker push, kubectl apply, ssh commands, sed edits, and git commit — without actually touching anything.

./deploy.sh --dry-run 1.4.0 staging
# Output:
# [DRY-RUN] docker build -f Dockerfile.staging . -t <registry>/<image>:1.4.0
# [DRY-RUN] docker push <registry>/<image>:1.4.0
# ...etc...

State Files Written

.env / .env.* (any depth-2 file matching)

NEXT_PUBLIC_APP_VERSION=1.4.0

The env-var name (NEXT_PUBLIC_APP_VERSION) is Next.js-specific — change the grep/sed patterns in version.sh to match your framework.

kubernetes/deploy.yaml (local & staging only)

All image tags :X.Y.Z are replaced with the new version via regex.

.version (repo root — written on every run)

CURRENT_VERSION=1.4.0
LAST_DEPLOY_ENV=staging
LAST_DEPLOY_AT=2026-08-03T14:03:11+04:00
LAST_DEPLOY_USER=pierre
GIT_BRANCH=main
GIT_COMMIT_SHA=a1b2c3d

This file is sourced on the next run to provide the OLD_VERSION baseline.


Docker Build Strategy

Each environment maps to a specific Dockerfile:

dockerfile_for_env() {
case "$1" in
local) echo "Dockerfile.local" ;;
staging) echo "Dockerfile.staging" ;;
production) echo "Dockerfile.prod" ;;
*) die "Unknown env '$1'" ;;
esac
}

The expected Dockerfiles per service directory — defined in build_all_images():

Service directoryRequired Dockerfiles
api/Dockerfile.local, Dockerfile.staging, Dockerfile.prod
www/Dockerfile.local, Dockerfile.staging, Dockerfile.prod
worker/Dockerfile.local, Dockerfile.staging, Dockerfile.prod

Image naming convention (customize user and image names in build_all_images()):

<registry-user>/<project>-api:<version>
<registry-user>/<project>-frontend:<version>
<registry-user>/<project>-worker:<version>

Complete File Reference

deploy.sh

#!/bin/bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"

# load lib
# shellcheck disable=SC1091
. "$REPO_ROOT/lib/index.sh"

# --- REMOTE HOSTS (customize per project) ---
STAGING_HOST="your-k8s-control-node"
PRODUCTION_HOST_1="your-vm-1"
PRODUCTION_HOST_2="your-vm-2"

parse_args() {
DRY_RUN=0

while [[ $# -gt 0 ]]; do
case "$1" in
-n | --dry-run)
DRY_RUN=1
shift
;;
-h | --help | help)
print_help
exit 0
;;
-*)
die "Unknown flag: $1"
;;
*)
break
;;
esac
done

NEW_VERSION="${1:-}"
shift || true
ENV_NAME="${1:-}"
shift || true

[ -z "${NEW_VERSION:-}" ] && die "Version tag not specified."
[ -z "${ENV_NAME:-}" ] && die "Environment not specified."
case "$ENV_NAME" in local | staging | production) ;; *) die "Invalid env '$ENV_NAME'. Allowed: local|staging|production." ;; esac

if [ -n "${1:-}" ]; then
die "Unknown argument: $1"
fi
}

run() {
parse_args "$@"

if [ -f .version ]; then
. .version
OLD_VERSION="${CURRENT_VERSION:-0.0.0}"
else
OLD_VERSION="0.0.0"
fi

echo "Old Version: $OLD_VERSION"
echo "New Version: $NEW_VERSION"
echo "Environment: $ENV_NAME"
echo "DRY_RUN=${DRY_RUN}"

UPDATE_KUBE="true"
[ "$ENV_NAME" = "production" ] && UPDATE_KUBE="false"

update_envs "$OLD_VERSION" "$NEW_VERSION" "$ENV_NAME" "$UPDATE_KUBE"

if [ "$ENV_NAME" = "local" ]; then
echo "==> Local build selected"
build_all_images "$NEW_VERSION" "$ENV_NAME"
exec_kube "local"
echo "Local deploy complete."
fi

if [ "$ENV_NAME" = "staging" ]; then
echo "==> Staging build selected"
build_all_images "$NEW_VERSION" "$ENV_NAME"
push_all_images "$NEW_VERSION"
exec_kube "$STAGING_HOST"
echo "Staging deploy complete."
fi

if [ "$ENV_NAME" = "production" ]; then
echo "==> Prod build selected"
build_all_images "$NEW_VERSION" "$ENV_NAME"
push_all_images "$NEW_VERSION"
systemd_update_remote_host "$PRODUCTION_HOST_1" "$NEW_VERSION"
systemd_update_remote_host "$PRODUCTION_HOST_2" "$NEW_VERSION"
echo "Production deploy complete."
fi

git_commit "$NEW_VERSION" "$ENV_NAME"
echo "Deployment complete"
}

run "$@"

lib/index.sh

#!/usr/bin/env bash
# Loads all *.sh modules in this directory (except itself), once.

if [ -n "${__LIB_INDEX_SH_SOURCED:-}" ]; then
return 0 2>/dev/null || exit 0
fi
readonly __LIB_INDEX_SH_SOURCED=1

_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"

while IFS= read -r -d '' mod; do
case "$(basename "$mod")" in
index.sh) continue ;;
esac
# shellcheck source=/dev/null
. "$mod"
done < <(find "$_LIB_DIR" -maxdepth 1 -type f -name '*.sh' -print0 | sort -z)

lib/helpers.sh

#!/usr/bin/env bash
: "${TOP_PID:=$$}"

die() {
local msg="${1:-"Unknown error"}"
echo "[ERROR] $msg" >&2
if [ -n "${USAGE:-}" ]; then echo "$USAGE" >&2; fi
if [ -n "${TOP_PID:-}" ] && kill -0 "$TOP_PID" 2>/dev/null; then
kill -s TERM "$TOP_PID"
else
exit 1
fi
}

pushd_quiet() {
local d="$1"
echo
echo "Changing working directory to ./$d..."
pushd "$d" &>/dev/null || die "Failed to enter $d."
}

popd_quiet() {
popd &>/dev/null || die "Failed to return to previous directory."
}

is_dry_run() {
case "${DRY_RUN:-0}" in
1 | true | TRUE | on | ON | yes | YES) return 0 ;;
*) return 1 ;;
esac
}

maybe() {
if is_dry_run; then
echo "[DRY-RUN] $*"
else
"$@"
fi
}

now_ts() {
date -Iseconds
}

lib/docker.sh

#!/usr/bin/env bash

dockerfile_for_env() {
case "$1" in
local) echo "Dockerfile.local" ;;
staging) echo "Dockerfile.staging" ;;
production) echo "Dockerfile.prod" ;;
*) die "Unknown env '$1' for Dockerfile mapping (expected: local|staging|production)" ;;
esac
}

ensure_dockerfile() {
local df="$1"
if [ ! -f "$df" ]; then
echo "[ERROR] Required Dockerfile not found: $df" >&2
echo " CWD: $(pwd)" >&2
echo " Available Dockerfile* here:" >&2
ls -1 Dockerfile* 2>/dev/null || true
die "Missing Dockerfile: $df"
fi
}

docker_build() {
local repo="$1" env="$2"
[ -z "$repo" ] && die "Repo required for docker build"
[ -z "$env" ] && die "Env required for docker build"
local df
df="$(dockerfile_for_env "$env")"
ensure_dockerfile "$df"
echo "Building image: $repo using $df"
maybe docker build -f "$df" . -t "$repo"
}

docker_push() {
local repo="$1"
[ -z "$repo" ] && die "Repo required for docker push"
maybe docker push "$repo"
}

# --- Customize: add/remove service directories and image names ---
build_all_images() {
local version="$1" env="$2"
[ -z "$version" ] && die "Version tag not specified."
[ -z "$env" ] && die "Env tag not specified."
local user="your-registry-user"

pushd_quiet "api"
docker_build "$user/<project>-api:$version" "$env"
popd_quiet
pushd_quiet "www"
docker_build "$user/<project>-frontend:$version" "$env"
popd_quiet
pushd_quiet "worker"
docker_build "$user/<project>-worker:$version" "$env"
popd_quiet
}

push_all_images() {
local version="$1"
[ -z "$version" ] && die "Version tag not specified."
local user="your-registry-user"
maybe docker_push "$user/<project>-api:$version"
maybe docker_push "$user/<project>-frontend:$version"
maybe docker_push "$user/<project>-worker:$version"
}

lib/kube.sh

#!/bin/bash

exec_kube() {
local remote_host="$1"
[ -z "$remote_host" ] && die "Remote host not specified."

if [ "$remote_host" = "local" ]; then
maybe rm -rf ~/kubernetes/<project>/*
maybe cp -r ./kubernetes/* ~/kubernetes/<project>/
maybe kubectl apply -k ~/kubernetes/<project> --force
else
maybe ssh "$remote_host" rm -rf ~/kubernetes/<project>/*
maybe scp -r -v ./kubernetes/* "$remote_host":~/kubernetes/<project>/
maybe ssh "$remote_host" 'kubectl apply -k ~/kubernetes/<project> --force'
fi
}

lib/systemd.sh

#!/bin/bash

SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=8)

_systemd_remote_update_one() {
local host="$1" svc="$2" new_tag="$3"
echo "INFO: running remote service update for ${svc}.service on HOST: ${host}"

maybe ssh "${SSH_OPTS[@]}" "$host" bash -s -- "$svc" "$new_tag" <<'REMOTE'
set -euo pipefail
svc="$1"
new_tag="$2"
env_file="/etc/default/${svc}"
if [ ! -f "$env_file" ]; then
echo "WARN: ${env_file} not found, skipping" >&2
exit 0
fi
echo "INFO: making backup ${env_file}.bak"
sudo cp -a "$env_file" "${env_file}.bak"
line="$(grep -m1 -E '^IMAGE=' "$env_file" || true)"
if [ -z "$line" ]; then
echo "WARN: No IMAGE= in ${env_file}; skipping"
else
image="${line#IMAGE=}"
if [[ "$image" == *:* ]]; then
repo="${image%:*}"
new_image="${repo}:${new_tag}"
else
new_image="${image}:${new_tag}"
fi
if [ "$image" = "$new_image" ]; then
echo "INFO: ${env_file} already at ${new_image}"
else
echo "UPDATE: ${env_file} IMAGE=$image -> IMAGE=$new_image"
sudo awk -v newimg="IMAGE=${new_image}" '
BEGIN { done=0 }
{
if (!done && $0 ~ /^IMAGE=/) { print newimg; done=1 }
else { print }
}
' "$env_file" | sudo tee "${env_file}.tmp" > /dev/null
sudo mv "${env_file}.tmp" "$env_file"
fi
fi
sudo systemctl daemon-reload
sudo systemctl restart "${svc}.service"
sudo systemctl --no-pager --full status "${svc}.service" -n 0 || true
echo "INFO: service reloaded ${svc}.service"
REMOTE
}

# --- Customize: list your systemd service names here ---
systemd_update_remote_host() {
local host="$1" new_tag="$2"
for svc in <project>-api <project>-frontend <project>-worker; do
_systemd_remote_update_one "$host" "$svc" "$new_tag"
done
}

lib/version.sh

#!/usr/bin/env bash

_git_branch() { git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "unknown"; }
_git_sha() { git rev-parse --short HEAD 2>/dev/null || echo "unknown"; }
_user_name() { id -un 2>/dev/null || whoami 2>/dev/null || echo "unknown"; }

_write_version_file() {
local new_version="$1" env="$2"
local ts user branch sha
ts="$(now_ts)"
user="$(_user_name)"
branch="$(_git_branch)"
sha="$(_git_sha)"

echo "Writing .version with:"
echo " CURRENT_VERSION=$new_version"
echo " LAST_DEPLOY_ENV=$env"
echo " LAST_DEPLOY_AT=$ts"
echo " LAST_DEPLOY_USER=$user"
echo " GIT_BRANCH=$branch"
echo " GIT_COMMIT_SHA=$sha"

if is_dry_run; then
echo "[DRY-RUN] > .version (not writing)"
else
cat >.version <<EOF
CURRENT_VERSION=$new_version
LAST_DEPLOY_ENV=$env
LAST_DEPLOY_AT=$ts
LAST_DEPLOY_USER=$user
GIT_BRANCH=$branch
GIT_COMMIT_SHA=$sha
EOF
fi
}

update_envs() {
echo "$@"
local old_version="$1" new_version="$2" env="$3" update_kube="${4:-false}"
[ -z "$new_version" ] && die "Usage: update_envs <old_version> <new_version> <env> [update_kube]"

echo "Updating .env files with APP_VERSION=$new_version..."
while IFS= read -r -d '' env_file; do
if grep -q "^APP_VERSION=" "$env_file"; then
maybe sed -i "s/^APP_VERSION=.*/APP_VERSION=$new_version/" "$env_file"
echo " - updated: $env_file"
fi
done < <(find . -mindepth 2 -maxdepth 2 -type f \( -name ".env" -o -name ".env.*" \) -print0)

if [ "$update_kube" = "true" ]; then
if [ -f kubernetes/deploy.yaml ]; then
echo "Updating kubernetes/deploy.yaml image tags: :$old_version -> :$new_version ..."
maybe sed -E -i "s/:[0-9]+\.[0-9]+\.[0-9]+/:${new_version}/g" kubernetes/deploy.yaml
else
echo "[WARN] kubernetes/deploy.yaml not found; skipping."
fi
fi

_write_version_file "$new_version" "$env"
}

lib/git.sh

#!/usr/bin/env bash

git_commit() {
version="$1"
env="$2"
echo "[INFO]: Running git commit for version $version to $env"
maybe git add .
maybe git commit -m "deploy: version $version to $env"
}

lib/help.sh

#!/usr/bin/env bash

print_help() {
cat <<'EOF'
Deploy Pipeline

USAGE
./deploy.sh <version> <env: local|staging|production> [options]

POSITIONAL ARGUMENTS
version Docker image tag to build/push/use (e.g. 1.4.0)
env One of:
- local : build all images with Dockerfile.local,
update .env files, bump kubernetes/deploy.yaml,
kubectl apply locally
- staging : build & push images (Dockerfile.staging),
update .env files, bump kubernetes/deploy.yaml,
kubectl apply on staging host via SSH
- production : build & push images (Dockerfile.prod),
update .env files, update bare-metal hosts
via SSH + systemd

OPTIONS
-n, --dry-run Print what would happen without making any changes
-h, --help, help Show this help and exit

BEHAVIOR
- Dockerfiles (required per service dir):
local -> Dockerfile.local
staging -> Dockerfile.staging
production -> Dockerfile.prod
The build will FAIL if the expected Dockerfile is missing.

- Files/State changed:
- .env / .env.* : APP_VERSION=<version> is updated (if present)
- kubernetes/deploy.yaml : image tags :<old> -> :<version> (local & staging only)
- .version : snapshot of the last run

- Remote hosts are hard-coded in the script — edit to match your infrastructure.

ENV VARS
DRY_RUN=1 Same effect as --dry-run

EXAMPLES
./deploy.sh 1.4.0 local
./deploy.sh 1.4.0 staging
./deploy.sh 1.4.0 production
./deploy.sh --dry-run 1.4.0 staging
DRY_RUN=1 ./deploy.sh 1.4.0 local

EXIT CODES
0 success
1+ failure (missing args, missing Dockerfile, command error, etc.)
EOF
}

Key Design Decisions

Robust script-relative sourcing

The deploy script uses BASH_SOURCE to resolve its own directory, so it works no matter where you invoke it from:

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
. "$REPO_ROOT/lib/index.sh"

No config files — the script IS the config

Remote hosts, Docker registry users, and service names are all hard-coded directly in the scripts. This is intentional for a single-project monorepo — there's no indirection to chase when debugging. To adapt this for your project, edit the values in deploy.sh (hosts), docker.sh (images), and systemd.sh (service names).

Atomic systemd env updates

The systemd module backs up /etc/default/<svc> before touching it, writes to a .tmp file, then mvs it into place. If the SSH connection drops mid-write, the original file is intact and the service keeps running on the old image.

Everything is dry-run-aware

Every docker build, docker push, kubectl apply, ssh, scp, sed, and git commit goes through maybe(). A single --dry-run flag gives you complete visibility into what the pipeline will do before it touches anything.