Skip to main content

Production Operations

Server and infrastructure maintenance for sanctumhq.gg.

Architecture Summary​

  • EC2 t3.medium (us-east-2) runs the app stack via Docker Compose
  • ECS Fargate Spot runs SimC worker on-demand
  • Route 53 manages DNS for sanctumhq.gg
  • Cloudflare Pages hosts docs at docs.sanctumhq.gg
  • SES sends transactional email from noreply@sanctumhq.gg
  • S3 stores uploads, game data icons, and sim results
  • SQS queues SimC simulation jobs

Connecting to the Server​

ssh -i ~/.ssh/sanctum-key.pem ubuntu@YOUR_ELASTIC_IP
cd ~/sanctum

Deploying Code Changes​

Standard Deploy (after pushing to master)​

ssh -i ~/.ssh/sanctum-key.pem ubuntu@YOUR_ELASTIC_IP
cd ~/sanctum
git pull origin master
docker compose -f docker-compose.yml -f docker-compose.prod.yml up --build -d

The --build flag rebuilds only containers whose source changed. Unchanged containers restart instantly.

Deploy API Only (skip GDL rebuild)​

docker compose -f docker-compose.yml -f docker-compose.prod.yml up --build -d api

Deploy Frontend Only​

docker compose -f docker-compose.yml -f docker-compose.prod.yml up --build -d web
docker compose -f docker-compose.yml -f docker-compose.prod.yml restart nginx

Rebuild Everything from Scratch​

docker compose -f docker-compose.yml -f docker-compose.prod.yml down
docker compose -f docker-compose.yml -f docker-compose.prod.yml up --build -d

This preserves database data (stored in Docker volumes). To also wipe data, add -v to the down command.

Database Operations​

Run Prisma Migrations​

Migrations run automatically on API container startup. To run manually:

docker compose exec api npx prisma migrate deploy

Check Migration Status​

docker compose exec api npx prisma migrate status

Connect to PostgreSQL Directly​

docker compose exec postgres psql -U sanctum_prod -d sanctum

For GDL databases:

docker compose exec gdl-postgres psql -U sanctum_prod -d sanctum_catalog
docker compose exec gdl-postgres psql -U sanctum_prod -d sanctum_characters

Manual Database Backup​

docker compose exec postgres pg_dump -U sanctum_prod sanctum > backup-$(date +%Y%m%d).sql

To back up to S3:

docker compose exec postgres pg_dump -U sanctum_prod sanctum | \
  aws s3 cp - s3://sanctum-app/backups/sanctum-$(date +%Y%m%d).sql

Restore from Backup​

docker compose exec -T postgres psql -U sanctum_prod sanctum < backup-20260313.sql

SSL Certificate Management​

Certbot runs as a container and auto-renews every 12 hours. To check status:

docker compose exec certbot certbot certificates

Force Renewal​

docker compose -f docker-compose.yml -f docker-compose.prod.yml run --rm certbot \
  renew --force-renewal
docker compose -f docker-compose.yml -f docker-compose.prod.yml restart nginx

First-Time SSL Setup​

If setting up SSL on a fresh server:

./scripts/init-ssl.sh admin@sanctumhq.gg

Monitoring​

View Logs​

# All services
docker compose logs -f

# Specific service
docker compose logs -f api
docker compose logs -f gdl
docker compose logs -f nginx

# Last 100 lines
docker compose logs --tail 100 api

Check Container Status​

docker compose ps

Check Disk Usage​

df -h
docker system df

Clean Up Docker Disk Space​

docker system prune -f
docker image prune -a -f

Check Memory and CPU​

free -h
htop

SimC Worker​

Update SimC Worker Image​

After code changes to services/simc/:

cd ~/sanctum

# Re-login to ECR (tokens expire after 12 hours)
aws ecr get-login-password --region us-east-2 | \
  docker login --username AWS --password-stdin \
  YOUR_ACCOUNT_ID.dkr.ecr.us-east-2.amazonaws.com

# Rebuild and push
docker build --target production \
  -t YOUR_ACCOUNT_ID.dkr.ecr.us-east-2.amazonaws.com/sanctum-simc-worker:latest \
  services/simc/

docker push YOUR_ACCOUNT_ID.dkr.ecr.us-east-2.amazonaws.com/sanctum-simc-worker:latest

New ECS tasks will automatically use the latest image.

Check SimC Job Status​

  • SQS Console: Check sim-jobs queue for pending messages
  • ECS Console: Check sanctum cluster for running/stopped tasks
  • CloudWatch Logs: Check /ecs/sanctum-simc-worker log group
  • Dead letter queue: Check sim-jobs-dlq for failed jobs

Environment Variables​

Editing Production Config​

nano ~/sanctum/apps/api/.env

After changing env vars, restart the API:

docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d api

Rotating Secrets​

Generate new secrets:

openssl rand -base64 32   # JWT_SECRET, SESSION_SECRET
openssl rand -hex 32      # ENCRYPTION_KEY

Update apps/api/.env, then restart the API. Note: rotating JWT_SECRET invalidates all existing user sessions.

DNS (Route 53)​

Current records:

RecordTypeValue
sanctumhq.ggAEC2 Elastic IP
www.sanctumhq.ggCNAMEsanctumhq.gg
docs.sanctumhq.ggCNAMEsanctum-docs.pages.dev
SES DKIM recordsCNAME(auto-generated)

Common Issues​

Container Won't Start​

docker compose logs api    # Check for errors
docker compose ps          # Check status
docker compose restart api # Try restarting

Out of Disk Space​

docker system prune -a -f  # Remove unused images
docker volume prune -f     # Remove unused volumes (careful!)

SSL Certificate Expired​

docker compose -f docker-compose.yml -f docker-compose.prod.yml run --rm certbot renew
docker compose -f docker-compose.yml -f docker-compose.prod.yml restart nginx

Can't SSH In​

Check that your IP hasn't changed -- the security group limits SSH to "My IP". Update the inbound rule in the EC2 Console if your IP changed.

API Health Check Failing​

docker compose exec api wget -qO- http://localhost:3001/api/v1/health

If it fails, check logs and ensure the database is healthy:

docker compose exec postgres pg_isready -U sanctum_prod -d sanctum

Cost Monitoring​

Check your AWS bill: Billing > Bills in the AWS Console. Expected ~$33-38/month:

  • EC2 t3.medium: ~$30
  • Route 53: $0.50
  • S3 + ECR: < $2
  • SQS + SES: Free tier
  • ECS Fargate Spot: $0-5 (per sim usage)