SDLC 05: Infrastructure Management and Release Procedures

Revision history: Updated June 2026 — 19-microservice architecture; Go/Python services; systemd; GitHub Actions workflows (ci.yml, deploy-staging.yml, deploy-production.yml, health-check.yml, db-backup.yml); branch strategy: main = staging, production = live.

---

1. Branch Strategy

The Pakashop branching model is designed for simplicity and continuous integration:

Branch	Environment	Purpose	Trigger
main	Staging	Integration & Pre-production QA	Push to main
production	Live	Production Platform	PR from main to production
feature/*	Preview	Local development & Vercel previews	PR to main
hotfix/*	Live	Critical production fixes	PR to production

---

2. Infrastructure Overview

Tier	Platform	Notes
Frontend	Vercel	Production branch: production; Staging branch: main
Backend (Staging)	AWS EC2 (Staging)	Tracks main branch; all 19 systemd services
Backend (Live)	AWS EC2 (Production)	Tracks production branch; all 19 systemd services
Database	RDS PostgreSQL	Separate instances for staging and production
Search	Meilisearch	Dedicated EC2 instance or container per environment
Redis	EC2 Redis	Per-environment Redis instance
CDN/DNS/WAF	Cloudflare	Full Strict TLS, OWASP WAF rules

---

3. EC2 Service Orchestration

All 19 backend services run under systemd.

3.1 systemd Services

Service	systemd Unit	Port	Language
API Gateway	pakashop-gateway.service	8000	Node.js
Backend API	pakashop-backend.service	3080	Node.js
Config Service	pakashop-config.service	3085	Node.js
Notifications	pakashop-notifications.service	3090	Node.js
Tracking	pakashop-tracking.service	3120	Node.js
Moderation	pakashop-moderation.service	3110	Python
Recommendations	pakashop-recommendations.service	3100	Python
Scheduler	pakashop-scheduler.service	3004	Node.js
Search	pakashop-search.service	3005	Go
Analytics	pakashop-analytics.service	3007	Go
Fraud	pakashop-fraud.service	3006	Node.js
Coupon	pakashop-coupon.service	3008	Node.js
Loyalty	pakashop-loyalty.service	3010	Node.js
WhatsApp	pakashop-whatsapp.service	3009	Node.js
Reports	pakashop-reports.service	3011	Node.js
Reconciliation	pakashop-reconciliation.service	3012	Node.js
Invoicing	pakashop-invoicing.service	3013	Node.js
Pricing	pakashop-pricing.service	3014	Node.js
Settlement	pakashop-settlement.service	3016	Node.js

3.2 Service Dependency Order

Services must start in dependency order. The deploy script handles this:

1. Redis, PostgreSQL, Meilisearch (infrastructure)
2. pakashop-config (feature flags needed by all)
3. pakashop-gateway (depends on config)
4. pakashop-backend, pakashop-search, pakashop-analytics (core services)
5. pakashop-notifications, pakashop-tracking, pakashop-moderation,
   pakashop-recommendations, pakashop-fraud, pakashop-coupon,
   pakashop-loyalty, pakashop-whatsapp, pakashop-reports,
   pakashop-reconciliation, pakashop-invoicing, pakashop-pricing,
   pakashop-settlement, pakashop-scheduler (supporting services)

---

4. CI/CD Pipelines (GitHub Actions)

4.1 CI Pipeline (ci.yml)

Triggered on every Pull Request to main or production:

name: CI
on:
  pull_request:
    branches: [main, production]

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env: { POSTGRES_PASSWORD: test, POSTGRES_DB: pakashop_test }
        ports: ['5432:5432']
      redis:
        image: redis:7
        ports: ['6379:6379']
      meilisearch:
        image: getmeili/meilisearch:v1.7
        env: { MEILI_MASTER_KEY: test_key }
        ports: ['7700:7700']
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - uses: actions/setup-go@v5
        with: { go-version: '1.22' }
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }

      # Node.js services
      - run: npm ci
      - run: npx prisma validate
      - run: npx prisma migrate deploy
      - run: npm run test:unit
      - run: npm run test:integration
      - run: npm run lint
      - run: npm audit --audit-level=moderate

      # Go services
      - run: cd services/search && go test ./...
      - run: cd services/analytics && go test ./...

      # Python services
      - run: cd services/moderation && pip install -r requirements.txt && pytest
      - run: cd services/recommendations && pip install -r requirements.txt && pytest

      # Frontend
      - run: cd frontend && npm ci && npm run test && npm run build

4.2 Staging Deployment (deploy-staging.yml)

Triggered on push to main:

name: Deploy Staging
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to Staging EC2
        uses: appleboy/ssh-action@v1
        with:
          host: ${{ secrets.EC2_STAGING_HOST }}
          username: deploy
          key: ${{ secrets.EC2_SSH_KEY }}
          script: |
            cd /opt/pakashop
            sudo ./scripts/deploy.sh staging

4.3 Production Deployment (deploy-production.yml)

Triggered on push to production:

name: Deploy Production
on:
  push:
    branches: [production]

jobs:
  backup:
    runs-on: ubuntu-latest
    steps:
      - name: Pre-deploy Database Backup
        run: |
          pg_dump ${{ secrets.PROD_DATABASE_URL }} > backup-$(date +%Y%m%d-%H%M%S).sql
          aws s3 cp backup-*.sql s3://pakashop-backups/production/

  deploy:
    needs: backup
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to Production EC2
        uses: appleboy/ssh-action@v1
        with:
          host: ${{ secrets.EC2_PROD_HOST }}
          username: deploy
          key: ${{ secrets.EC2_SSH_KEY }}
          script: |
            cd /opt/pakashop
            sudo ./scripts/deploy.sh production

4.4 Health Check (health-check.yml)

Runs every 15 minutes:

name: Health Check
on:
  schedule:
    - cron: '*/15 * * * *'

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - name: Check Production Health
        run: |
          curl -f https://pakashop.store/api/v1/health || exit 1
          curl -f https://pakashop.store/api/v1/health/ready || exit 1
      - name: Check Staging Health
        run: |
          curl -f https://staging.pakashop.store/api/v1/health || exit 1

4.5 Database Backup (db-backup.yml)

Nightly automated backup:

name: Database Backup
on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM UTC daily

jobs:
  backup:
    runs-on: ubuntu-latest
    steps:
      - name: Backup Production Database
        run: |
          TIMESTAMP=$(date +%Y%m%d-%H%M%S)
          pg_dump ${{ secrets.PROD_DATABASE_URL }} | gzip > pakashop-prod-${TIMESTAMP}.sql.gz
          aws s3 cp pakashop-prod-${TIMESTAMP}.sql.gz s3://pakashop-backups/daily/
          # Retain last 30 days
          aws s3 ls s3://pakashop-backups/daily/ | sort | head -n -30 | awk '{print $4}' | xargs -I {} aws s3 rm s3://pakashop-backups/daily/{}

---

5. Deployment Flow

5.1 Frontend (Vercel)

• Staging: Pushes to main are automatically deployed to staging.pakashop.store.
• Production: Merges into production are automatically deployed to pakashop.store.
• Preview: feature/* branches get preview deployments at <branch>.pakashop-pr.vercel.app.

5.2 Backend (EC2)

Deployment is orchestrated via GitHub Actions using the deploy.sh script on the target host.

Deployment Script (scripts/deploy.sh):

#!/bin/bash
set -e

ENV=$1  # staging or production
BRANCH=$(git rev-parse --abbrev-ref HEAD)

echo "Deploying to $ENV (branch: $BRANCH)"

# Pull latest code
git pull origin $BRANCH

# Node.js services
for service in gateway backend config notifications tracking scheduler fraud coupon loyalty whatsapp reports reconciliation invoicing pricing settlement; do
  echo "Deploying $service..."
  cd services/$service
  npm ci --production
  npx prisma migrate deploy  # if applicable
  cd ../..
  sudo systemctl restart pakashop-$service
  sleep 2
  sudo systemctl is-active --quiet pakashop-$service || exit 1
done

# Go services
for service in search analytics; do
  echo "Deploying $service..."
  cd services/$service
  go build -o bin/$service ./src
  cd ../..
  sudo systemctl restart pakashop-$service
  sleep 2
  sudo systemctl is-active --quiet pakashop-$service || exit 1
done

# Python services
for service in moderation recommendations; do
  echo "Deploying $service..."
  cd services/$service
  source venv/bin/activate
  pip install -r requirements.txt
  cd ../..
  sudo systemctl restart pakashop-$service
  sleep 2
  sudo systemctl is-active --quiet pakashop-$service || exit 1
done

# Reload nginx
sudo systemctl reload nginx

echo "Deployment to $ENV completed successfully"

---

6. Release Procedures

6.1 Standard Release

1. Complete feature development on feature/*.
2. Open PR to main. After review and CI pass, merge.
3. Verify changes on the Staging environment (main branch).
4. Run E2E tests on staging (Playwright).
5. Run performance tests on staging (k6).
6. Once verified, open PR from main to production.
7. Merge PR to trigger live deployment (includes pre-deploy DB backup).

6.2 Hotfix Release

1. Branch from production: hotfix/<description>.
2. Apply fix and verify.
3. Merge directly to production for immediate live deployment.
4. Important: Immediately merge production back into main to keep environments synchronized.

---

7. Rollback Procedure

7.1 Code Rollback

1. Revert: git revert HEAD on the corresponding branch (main or production).
2. Deploy: Push the revert to trigger the automated deployment.
3. Verify: Check pakashop-status.sh and logs to ensure stability.

7.2 Database Rollback

If a migration caused data issues:

1. Restore the most recent RDS snapshot from S3.
2. Manually resolve the migration state:

   npx prisma migrate resolve --rolled-back <migration_name>

7.3 Service Rollback

For individual service failures:

# Roll back a single service to previous commit
cd services/backend
git checkout production~1 -- .
npm ci --production
sudo systemctl restart pakashop-backend

---

For internal use only. Do not distribute outside Pakashop engineering.