Skip to main content

Production Operations

Server and infrastructure maintenance for sanctumhq.gg.

Architecture Summary​

  • EC2 t3.medium (us-east-2) runs the app stack via Docker Compose
  • ECS Fargate Spot runs SimC worker on-demand
  • Route 53 manages DNS for sanctumhq.gg
  • Cloudflare Pages hosts docs at docs.sanctumhq.gg
  • SES sends transactional email from noreply@sanctumhq.gg
  • S3 stores uploads, game data icons, and sim results
  • SQS queues SimC simulation jobs

Connecting to the Server​

ssh -i ~/.ssh/sanctum-key.pem ubuntu@YOUR_ELASTIC_IP
cd ~/sanctum

Deploying Code Changes​

Pushing to master triggers the GitHub Actions deploy workflow automatically. The pipeline:

  1. Waits for lint + unit tests + E2E tests to pass
  2. SSHes into the EC2 instance
  3. Pulls the latest code
  4. Rebuilds changed containers
  5. Runs Prisma migrations
  6. Cleans up old images

You can also trigger it manually via the Actions tab → Deploy → Run workflow.

Required GitHub Secrets​

Set these in Settings → Secrets and variables → Actions:

SecretDescription
EC2_HOSTElastic IP or hostname of the EC2 instance
EC2_USERSSH user (usually ubuntu)
EC2_SSH_KEYPrivate key contents for SSH access
DEPLOY_PATHProject directory on server (default: ~/sanctum)

GitHub Environment​

Create a production environment in Settings → Environments with optional protection rules (required reviewers, wait timer) for extra safety.

Manual Deploy (via script)​

ssh -i ~/.ssh/sanctum-key.pem ubuntu@YOUR_ELASTIC_IP
cd ~/sanctum
./scripts/deploy.sh

Use --skip-migrations if you know there are no pending migrations.

Deploy Individual Services​

# API only (skip GDL rebuild)
docker compose -f docker-compose.yml -f docker-compose.prod.yml up --build -d api

# Frontend only
docker compose -f docker-compose.yml -f docker-compose.prod.yml up --build -d web
docker compose -f docker-compose.yml -f docker-compose.prod.yml restart nginx

Rebuild Everything from Scratch​

docker compose -f docker-compose.yml -f docker-compose.prod.yml down
docker compose -f docker-compose.yml -f docker-compose.prod.yml up --build -d

This preserves database data (stored in Docker volumes). To also wipe data, add -v to the down command.

Database Operations​

Run Prisma Migrations​

Migrations run automatically on API container startup. To run manually:

docker compose exec api npx prisma migrate deploy

Check Migration Status​

docker compose exec api npx prisma migrate status

Connect to PostgreSQL Directly​

docker compose exec postgres psql -U sanctum_prod -d sanctum

For GDL databases:

docker compose exec gdl-postgres psql -U sanctum_prod -d sanctum_catalog
docker compose exec gdl-postgres psql -U sanctum_prod -d sanctum_characters

Manual Database Backup​

docker compose exec postgres pg_dump -U sanctum_prod sanctum > backup-$(date +%Y%m%d).sql

To back up to S3:

docker compose exec postgres pg_dump -U sanctum_prod sanctum | \
  aws s3 cp - s3://sanctum-app/backups/sanctum-$(date +%Y%m%d).sql

Restore from Backup​

docker compose exec -T postgres psql -U sanctum_prod sanctum < backup-20260313.sql

pgAdmin (Database GUI)​

pgAdmin runs in production at https://sanctumhq.gg/pgadmin/. It connects to both the main Sanctum database and the GDL (Game Data Library) database.

First Login​

Use the credentials from your environment variables:

  • Email: PGADMIN_EMAIL (default: admin@sanctumhq.gg)
  • Password: PGADMIN_PASSWORD

Both database servers are pre-configured — just enter the Postgres password when connecting to each.

Environment Variables​

Set these in the server environment (or .env):

VariableDescription
PGADMIN_EMAILLogin email (default: admin@sanctumhq.gg)
PGADMIN_PASSWORDRequired — login password

Updating Server Definitions​

Server connection details live in docker/pgadmin/servers.json. To add a new database, edit that file and redeploy.

SSL Certificate Management​

Certbot runs as a container and auto-renews every 12 hours. To check status:

docker compose exec certbot certbot certificates

Force Renewal​

docker compose -f docker-compose.yml -f docker-compose.prod.yml run --rm certbot \
  renew --force-renewal
docker compose -f docker-compose.yml -f docker-compose.prod.yml restart nginx

First-Time SSL Setup​

If setting up SSL on a fresh server:

./scripts/init-ssl.sh admin@sanctumhq.gg

Monitoring​

View Logs​

# All services
docker compose logs -f

# Specific service
docker compose logs -f api
docker compose logs -f gdl
docker compose logs -f nginx

# Last 100 lines
docker compose logs --tail 100 api

Check Container Status​

docker compose ps

Check Disk Usage​

df -h
docker system df

Clean Up Docker Disk Space​

docker system prune -f
docker image prune -a -f

Check Memory and CPU​

free -h
htop

SimC Worker​

Update SimC Worker Image​

After code changes to services/simc/:

cd ~/sanctum

# Re-login to ECR (tokens expire after 12 hours)
aws ecr get-login-password --region us-east-2 | \
  docker login --username AWS --password-stdin \
  YOUR_ACCOUNT_ID.dkr.ecr.us-east-2.amazonaws.com

# Rebuild and push
docker build --target production \
  -t YOUR_ACCOUNT_ID.dkr.ecr.us-east-2.amazonaws.com/sanctum-simc-worker:latest \
  services/simc/

docker push YOUR_ACCOUNT_ID.dkr.ecr.us-east-2.amazonaws.com/sanctum-simc-worker:latest

New ECS tasks will automatically use the latest image.

Check SimC Job Status​

  • SQS Console: Check sim-jobs queue for pending messages
  • ECS Console: Check sanctum cluster for running/stopped tasks
  • CloudWatch Logs: Check /ecs/sanctum-simc-worker log group
  • Dead letter queue: Check sim-jobs-dlq for failed jobs

Environment Variables (API)​

Editing Production Config​

nano ~/sanctum/apps/api/.env

After changing env vars, restart the API:

docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d api

Rotating Secrets​

Generate new secrets:

openssl rand -base64 32   # JWT_SECRET, SESSION_SECRET
openssl rand -hex 32      # ENCRYPTION_KEY

Update apps/api/.env, then restart the API. Note: rotating JWT_SECRET invalidates all existing user sessions.

DNS (Route 53)​

Current records:

RecordTypeValue
sanctumhq.ggAEC2 Elastic IP
www.sanctumhq.ggCNAMEsanctumhq.gg
docs.sanctumhq.ggCNAMEsanctum-docs.pages.dev
SES DKIM recordsCNAME(auto-generated)

Common Issues​

Container Won't Start​

docker compose logs api    # Check for errors
docker compose ps          # Check status
docker compose restart api # Try restarting

Out of Disk Space​

docker system prune -a -f  # Remove unused images
docker volume prune -f     # Remove unused volumes (careful!)

SSL Certificate Expired​

docker compose -f docker-compose.yml -f docker-compose.prod.yml run --rm certbot renew
docker compose -f docker-compose.yml -f docker-compose.prod.yml restart nginx

Can't SSH In​

Check that your IP hasn't changed -- the security group limits SSH to "My IP". Update the inbound rule in the EC2 Console if your IP changed.

API Health Check Failing​

docker compose exec api wget -qO- http://localhost:3001/api/v1/health

If it fails, check logs and ensure the database is healthy:

docker compose exec postgres pg_isready -U sanctum_prod -d sanctum

Cost Monitoring​

Check your AWS bill: Billing > Bills in the AWS Console. Expected ~$33-38/month:

  • EC2 t3.medium: ~$30
  • Route 53: $0.50
  • S3 + ECR: < $2
  • SQS + SES: Free tier
  • ECS Fargate Spot: $0-5 (per sim usage)