CI/CD Pipeline

Related docs: SDLC/DEPLOYMENT · Hosting Infrastructure · Microservices

---

1. Branching Strategy

The repository follows a simplified two-tier deployment model:

• main: The integration branch. Corresponds to the Staging environment.
• production: The release branch. Corresponds to the Live environment.
• feature/*: Individual feature development.
• hotfix/*: Critical production fixes.

---

2. CI Pipeline (ci.yml)

Triggered on every Pull Request to main or production:

name: CI
on:
  pull_request:
    branches: [main, production]

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: test
          POSTGRES_DB: pakashop_test
        ports: ['5432:5432']
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      redis:
        image: redis:7
        ports: ['6379:6379']
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      meilisearch:
        image: getmeili/meilisearch:v1.7
        env:
          MEILI_MASTER_KEY: test_key
        ports: ['7700:7700']

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Setup Go
        uses: actions/setup-go@v5
        with:
          go-version: '1.22'

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      # Node.js services
      - name: Install Node dependencies
        run: npm ci

      - name: Validate Prisma schema
        run: npx prisma validate

      - name: Run database migrations
        run: npx prisma migrate deploy
        env:
          DATABASE_URL: postgres://postgres:test@localhost:5432/pakashop_test

      - name: Run unit tests
        run: npm run test:unit

      - name: Run integration tests
        run: npm run test:integration
        env:
          DATABASE_URL: postgres://postgres:test@localhost:5432/pakashop_test
          REDIS_URL: redis://localhost:6379
          MEILISEARCH_URL: http://localhost:7700

      - name: Run linting
        run: npm run lint

      - name: Security audit
        run: npm audit --audit-level=moderate

      # Go services
      - name: Test search service
        run: cd services/search && go test ./...

      - name: Test analytics service
        run: cd services/analytics && go test ./...

      # Python services
      - name: Test moderation service
        run: |
          cd services/moderation
          python -m venv venv
          source venv/bin/activate
          pip install -r requirements.txt
          pytest

      - name: Test recommendations service
        run: |
          cd services/recommendations
          python -m venv venv
          source venv/bin/activate
          pip install -r requirements.txt
          pytest

      # Frontend
      - name: Install frontend dependencies
        run: cd frontend && npm ci

      - name: Run frontend tests
        run: cd frontend && npm run test

      - name: Build frontend
        run: cd frontend && npm run build

---

3. CD Pipeline — Frontend (Vercel)

Vercel is configured with two primary environments:

Environment	Tracking Branch	Domain
Production	production	pakashop.store
Staging	main	staging.pakashop.store
Preview	feature/*	<branch>.pakashop-pr.vercel.app

---

4. CD Pipeline — Backend (EC2 via GitHub Actions SSH)

4.1 Staging Deployment (deploy-staging.yml)

name: Deploy Staging
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Deploy to Staging EC2
        uses: appleboy/ssh-action@v1
        with:
          host: ${{ secrets.EC2_STAGING_HOST }}
          username: deploy
          key: ${{ secrets.EC2_SSH_KEY }}
          script: |
            cd /opt/pakashop
            sudo ./scripts/deploy.sh staging

4.2 Production Deployment (deploy-production.yml)

name: Deploy Production
on:
  push:
    branches: [production]

jobs:
  backup:
    runs-on: ubuntu-latest
    steps:
      - name: Pre-deploy Database Backup
        run: |
          TIMESTAMP=$(date +%Y%m%d-%H%M%S)
          pg_dump ${{ secrets.PROD_DATABASE_URL }} | gzip > pakashop-prod-${TIMESTAMP}.sql.gz
          aws s3 cp pakashop-prod-${TIMESTAMP}.sql.gz s3://pakashop-backups/production/
          # Retain last 30 days
          aws s3 ls s3://pakashop-backups/production/ | sort | head -n -30 | awk '{print $4}' | xargs -I {} aws s3 rm s3://pakashop-backups/production/{}

  deploy:
    needs: backup
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Deploy to Production EC2
        uses: appleboy/ssh-action@v1
        with:
          host: ${{ secrets.EC2_PROD_HOST }}
          username: deploy
          key: ${{ secrets.EC2_SSH_KEY }}
          script: |
            cd /opt/pakashop
            sudo ./scripts/deploy.sh production

4.3 Health Check (health-check.yml)

name: Health Check
on:
  schedule:
    - cron: '*/15 * * * *'

jobs:
  check-production:
    runs-on: ubuntu-latest
    steps:
      - name: Check Production Health
        run: |
          curl -f https://pakashop.store/api/v1/health || exit 1
          curl -f https://pakashop.store/api/v1/health/ready || exit 1

      - name: Check Staging Health
        run: |
          curl -f https://staging.pakashop.store/api/v1/health || exit 1
          curl -f https://staging.pakashop.store/api/v1/health/ready || exit 1

      - name: Notify on Failure
        if: failure()
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} \
            -H 'Content-Type: application/json' \
            -d '{"text":"🚨 Health check failed for Pakashop"}'

4.4 Database Backup (db-backup.yml)

name: Database Backup
on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM UTC daily

jobs:
  backup:
    runs-on: ubuntu-latest
    steps:
      - name: Backup Production Database
        run: |
          TIMESTAMP=$(date +%Y%m%d-%H%M%S)
          pg_dump ${{ secrets.PROD_DATABASE_URL }} | gzip > pakashop-prod-${TIMESTAMP}.sql.gz
          aws s3 cp pakashop-prod-${TIMESTAMP}.sql.gz s3://pakashop-backups/daily/

      - name: Backup Staging Database
        run: |
          TIMESTAMP=$(date +%Y%m%d-%H%M%S)
          pg_dump ${{ secrets.STAGING_DATABASE_URL }} | gzip > pakashop-staging-${TIMESTAMP}.sql.gz
          aws s3 cp pakashop-staging-${TIMESTAMP}.sql.gz s3://pakashop-backups/staging/

      - name: Cleanup Old Backups
        run: |
          # Retain last 30 days for daily backups
          aws s3 ls s3://pakashop-backups/daily/ | sort | head -n -30 | awk '{print $4}' | xargs -I {} aws s3 rm s3://pakashop-backups/daily/{}
          aws s3 ls s3://pakashop-backups/staging/ | sort | head -n -30 | awk '{print $4}' | xargs -I {} aws s3 rm s3://pakashop-backups/staging/{}

4.5 Security Scan (security-scan.yml)

name: Security Scan
on:
  pull_request:
    branches: [main, production]
  schedule:
    - cron: '0 0 * * 0'  # Weekly on Sunday

jobs:
  zap-baseline:
    runs-on: ubuntu-latest
    steps:
      - name: OWASP ZAP Baseline Scan
        uses: zaproxy/action-baseline@v0.12.0
        with:
          target: 'https://staging.pakashop.store'
          rules_file_name: '.zap/rules.tsv'

  npm-audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm audit --audit-level=moderate

---

5. Deployment Script (scripts/deploy.sh)

The deploy.sh script on the target host handles:

#!/bin/bash
set -e

ENV=$1
BRANCH=$(git rev-parse --abbrev-ref HEAD)

echo "=== Deploying to $ENV (branch: $BRANCH) ==="

# Validate environment
if [[ "$ENV" != "staging" && "$ENV" != "production" ]]; then
  echo "Error: Environment must be 'staging' or 'production'"
  exit 1
fi

# Pull latest code
git fetch origin
git reset --hard origin/$BRANCH

# Node.js services - install and restart
NODE_SERVICES=(
  "gateway"
  "backend"
  "config"
  "notifications"
  "tracking"
  "scheduler"
  "fraud"
  "coupon"
  "loyalty"
  "whatsapp"
  "reports"
  "reconciliation"
  "invoicing"
  "pricing"
  "settlement"
)

for service in "${NODE_SERVICES[@]}"; do
  echo "--- Deploying pakashop-$service ---"
  cd services/$service
  npm ci --production

  # Run Prisma migrations if applicable
  if [ -f "prisma/schema.prisma" ]; then
    npx prisma migrate deploy
  fi

  cd ../..
  sudo systemctl restart pakashop-$service
  sleep 2

  # Verify service started
  if ! sudo systemctl is-active --quiet pakashop-$service; then
    echo "ERROR: pakashop-$service failed to start"
    exit 1
  fi
  echo "✓ pakashop-$service is active"
done

# Go services - build and restart
GO_SERVICES=("search" "analytics")

for service in "${GO_SERVICES[@]}"; do
  echo "--- Deploying pakashop-$service ---"
  cd services/$service
  go build -o bin/$service ./src
  cd ../..
  sudo systemctl restart pakashop-$service
  sleep 2

  if ! sudo systemctl is-active --quiet pakashop-$service; then
    echo "ERROR: pakashop-$service failed to start"
    exit 1
  fi
  echo "✓ pakashop-$service is active"
done

# Python services - install and restart
PYTHON_SERVICES=("moderation" "recommendations")

for service in "${PYTHON_SERVICES[@]}"; do
  echo "--- Deploying pakashop-$service ---"
  cd services/$service
  source venv/bin/activate
  pip install -r requirements.txt
  cd ../..
  sudo systemctl restart pakashop-$service
  sleep 2

  if ! sudo systemctl is-active --quiet pakashop-$service; then
    echo "ERROR: pakashop-$service failed to start"
    exit 1
  fi
  echo "✓ pakashop-$service is active"
done

# Reload nginx
sudo systemctl reload nginx

echo "=== Deployment to $ENV completed successfully ==="

---

6. Rollback Procedure

6.1 Manual Rollback

If a deployment fails:

1. Revert the triggering commit on main or production.
2. The CD pipeline will automatically redeploy the previous stable state.

git revert HEAD
git push origin production

6.2 Database Rollback

If a migration caused data issues:

1. Restore the most recent RDS snapshot from S3.
2. Manually resolve the migration state:

   npx prisma migrate resolve --rolled-back <migration_name>

6.3 Service Rollback

For individual service failures:

cd services/backend
git checkout production~1 -- .
npm ci --production
sudo systemctl restart pakashop-backend

---

7. Deleting Legacy Branches

The old staging branch is no longer used. To clean up:

git push origin --delete staging
git branch -d staging

---

For internal use only. Do not distribute outside Pakashop engineering.