Email Service — VM Deploy Runbook

Status: 🟢 Active (mirror pola firebase + whatsapp) Owner: Backend Lead Latest: 2026-05-29 — Phase 4 DEPLOYED & VERIFIED 2026-05-29 15:50 WIB (ACCELERATED). core_api/internal/email/ DIHAPUS. Single path: emailservice.Client → HTTP. Phase 3 soak background Day 5/14 PASS ongoing.

Dokumen terkait:

architecture.md — desain endpoint + persistence + observability
extraction-plan.md — Phase -1 sampai Phase 4 plan
email-service-status.md — phase tracker visual
services/firebase/phase4-readiness.md — pola sister service yang ditiru
services/whatsapp/deploy-runbook.md — pola sister service
services/email_service/README.md — service-level runbook

1. VM topology

Item	Value
Host	`ptikn3-vm.cluster-vps.dalang.io`
User / Group	`enalfarid` / `enalfarid`
Folder root	`/home/enalfarid/kesles_merchant/merchant_email/`
Binary nama	`email-service` (Linux ELF amd64 stripped, ~8.9 MB)
Env file	`.env.production` (chmod 644, gitignored)
Systemd unit	`/etc/systemd/system/email-service.service`
Listen port	`127.0.0.1:8094` (loopback internal, dipakai core_api)
Logs	`sudo journalctl -u email-service [-f] [--since="1 hour ago"]`
Outbound	SMTP submission port 587 ke `smtp.gmail.com` + Postgres ke DB host internal

Pattern folder + systemd unit identik dengan merchant_firebase/ + merchant_whatsapp/ supaya operator tidak belajar layout baru per service.

2. Initial deploy (first-time setup)

2.1 Build binary di laptop

cd ~/Macbook\ pro/development/kesles_merchant/services/email_service

# Cross-compile Linux amd64 stripped
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
  go build \
    -trimpath \
    -ldflags="-s -w" \
    -o dist/email-service \
    ./cmd/server

# Verifikasi ukuran (~8-12 MB)
ls -lh dist/email-service
file dist/email-service  # → ELF 64-bit LSB executable, x86-64, statically linked

Atau pakai helper script:

./scripts/build_email_service.sh linux-prod   # → dist/email-service Linux ELF amd64 stripped

2.2 Create folder + env di VM

ssh enalfarid@ptikn3-vm.cluster-vps.dalang.io
mkdir -p ~/kesles_merchant/merchant_email/
cd ~/kesles_merchant/merchant_email/

# Copy template lalu isi credential
cp ~/kesles_merchant/services/email_service/.env.example .env.production
chmod 644 .env.production
nano .env.production

Env wajib di production (kalau salah satu kosong, service refuse start via validateConfig post G3 2026-05-25):

Env	Sumber
`APP_ENV=production`	static
`APP_PORT=8094`	static (loopback)
`INTERNAL_NOTIFICATION_API_KEY`	rotate sync dengan caller `EMAIL_SERVICE_API_KEY` di core_api
`POSTGRES_DSN`	`postgres://kesles:...@<DB_HOST>:5432/db_kesles_merchant?sslmode=disable`
`SMTP_HOST=smtp.gmail.com`	static (atau provider lain post-migration)
`SMTP_PORT=587`	static (submission port, STARTTLS)
`SMTP_USERNAME`	Gmail address sender
`SMTP_PASSWORD`	Gmail App Password (16-char, no spaces) — bukan password account
`SMTP_FROM_NAME=Kesles Merchant`	static
`SMTP_FROM_EMAIL`	display "From" header (biasanya sama dengan `SMTP_USERNAME`)
`EMAIL_RETRY_MAX`	optional, default 3
`EMAIL_RETRY_INITIAL_BACKOFF_MS`	optional, default 500
`EMAIL_RETRY_MAX_BACKOFF_MS`	optional, default 10000

Generate Gmail App Password: Google Account → Security → 2FA → App passwords → Generate. 16-char output (mis. abcd efgh ijkl mnop) — paste tanpa spasi ke SMTP_PASSWORD.

2.3 SCP binary

# Dari laptop
scp dist/email-service enalfarid@ptikn3-vm.cluster-vps.dalang.io:~/kesles_merchant/merchant_email/

# Verifikasi di VM
ssh enalfarid@ptikn3-vm.cluster-vps.dalang.io 'ls -lh ~/kesles_merchant/merchant_email/email-service'

2.4 Install systemd unit

Buat /etc/systemd/system/email-service.service:

[Unit]
Description=Kesles Merchant - Email Service
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=simple
User=enalfarid
Group=enalfarid
WorkingDirectory=/home/enalfarid/kesles_merchant/merchant_email
EnvironmentFile=/home/enalfarid/kesles_merchant/merchant_email/.env.production
ExecStart=/home/enalfarid/kesles_merchant/merchant_email/email-service
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal
SyslogIdentifier=email-service

# Hardening (selaras firebase + whatsapp)
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=read-only
PrivateTmp=true
ReadWritePaths=/home/enalfarid/kesles_merchant/merchant_email

[Install]
WantedBy=multi-user.target

Enable + start:

sudo systemctl daemon-reload
sudo systemctl enable email-service
sudo systemctl start email-service
sudo systemctl status email-service     # Expected: active (running)

2.5 Smoke test post-deploy

# 1. Liveness
curl -s http://127.0.0.1:8094/health | jq
# Expected: {"service":"email-service","status":"ok","smtp_configured":true,"postgres_configured":true,"app_env":"production"}

# 2. Readiness (DB ping + SMTP config check)
curl -s http://127.0.0.1:8094/ready | jq
# Expected: {"service":"email-service","status":"ready","checks":{"smtp":"ok","postgres":"ok"},"app_env":"production"}

# 3. Internal auth wall (tanpa header → 401)
curl -s -X POST http://127.0.0.1:8094/internal/email/send \
  -H "Content-Type: application/json" \
  -d '{"template":"otp","to_email":"test@example.com","payload":{"code":"000000"}}'
# Expected: {"error":"unauthorized"}

# 4. Internal endpoint dengan header (test ke email kamu sendiri, JANGAN customer)
KEY=$(grep "^INTERNAL_NOTIFICATION_API_KEY=" .env.production | cut -d= -f2)
curl -s -X POST http://127.0.0.1:8094/internal/email/send \
  -H "Content-Type: application/json" \
  -H "X-Internal-API-Key: $KEY" \
  -d '{"template":"otp","to_email":"<your-email@kesles.com>","payload":{"code":"123456"}}'
# Expected: {"status":"sent","template":"otp","message_id":"<uuid>"}

# 5. Lookup audit row
curl -s http://127.0.0.1:8094/internal/email/messages/<uuid> \
  -H "X-Internal-API-Key: $KEY" | jq

3. Re-deploy (update binary)

# 1. Build di laptop
cd ~/Macbook\ pro/development/kesles_merchant/services/email_service
./scripts/build_email_service.sh linux-prod

# 2. SCP (overwrite binary)
scp dist/email-service enalfarid@ptikn3-vm.cluster-vps.dalang.io:~/kesles_merchant/merchant_email/

# 3. Restart service (zero-downtime kurang dari 2 detik)
ssh enalfarid@ptikn3-vm.cluster-vps.dalang.io 'sudo systemctl restart email-service'

# 4. Verify (jangan skip — kalau startup gagal validateConfig, systemd restart loop)
ssh enalfarid@ptikn3-vm.cluster-vps.dalang.io \
  'sudo systemctl status email-service && curl -s http://127.0.0.1:8094/ready | jq'

Selama restart, ada window ~2 detik service tidak respond. Phase 4 DONE (2026-05-29): tidak ada lagi fallback — restart window jadi hard-block. Email yang fire saat window itu akan error. Pertimbangkan blue-green deploy (2 instance + LB) kalau traffic >100/min.

4. Env rotation

4.1 Rotate `INTERNAL_NOTIFICATION_API_KEY` (paired dengan core_api)

Service ini + caller core_api harus rotate sekaligus supaya tidak ada window auth mismatch.

# 1. Generate new key (32 char minimum)
NEW_KEY=$(openssl rand -base64 48 | tr -d '/+=' | head -c 48)

# 2. Di VM email: update .env.production
ssh enalfarid@ptikn3-vm.cluster-vps.dalang.io
nano ~/kesles_merchant/merchant_email/.env.production
# → INTERNAL_NOTIFICATION_API_KEY=<NEW_KEY>

# 3. Di VM core_api: update .env.production
nano ~/kesles_merchant/merchant_core_api/.env.production
# → EMAIL_SERVICE_API_KEY=<NEW_KEY> (same value)

# 4. Restart kedua service hampir bersamaan
sudo systemctl restart email-service && sudo systemctl restart merchant-core-api

# 5. Verify dengan trigger email dari dashboard (mis. password reset)

Window mismatch ~5 detik antara dua restart. Phase 4 DONE (2026-05-29): tidak ada lagi fallback ke direct SMTP. Email yang fire saat window mismatch akan error (auth fail). Lakukan restart hampir bersamaan untuk minimasi window.

4.2 Rotate `SMTP_PASSWORD` (Gmail App Password 90-hari)

Gmail App Password punya 90-hari rotation policy. Procedure:

# 1. Generate App Password baru di Google Account
#    Security → 2FA → App passwords → Generate
#    Copy 16-char output (mis. "abcd efgh ijkl mnop")

# 2. Update .env.production di VM (paste tanpa spasi)
ssh enalfarid@ptikn3-vm.cluster-vps.dalang.io
nano ~/kesles_merchant/merchant_email/.env.production
# → SMTP_PASSWORD=abcdefghijklmnop

# 3. Restart
sudo systemctl restart email-service

# 4. Smoke test send OTP ke email kamu
KEY=$(grep "^INTERNAL_NOTIFICATION_API_KEY=" .env.production | cut -d= -f2)
curl -s -X POST http://127.0.0.1:8094/internal/email/send \
  -H "X-Internal-API-Key: $KEY" \
  -d '{"template":"otp","to_email":"<your-email>","payload":{"code":"999999"}}'

# 5. Revoke App Password lama di Google Account UI (jangan biarkan menggantung)

Kalau App Password expire tanpa rotation, slog event email_send akan emit error_class=smtp_auth (post G4 regex ordering fix). Alert threshold di §5 Monitoring trigger immediate.

4.3 Rotate ke provider transactional (future)

Email_service sudah punya Provider interface di internal/email/provider.go. Migration ke SendGrid / AWS SES / Postmark = tambah provider implementation + swap di NewSender. Detail di extraction-plan.md roadmap.

5. Monitoring & alerting

5.1 Log stream

# Real-time tail
sudo journalctl -u email-service -f

# Hari ini
sudo journalctl -u email-service --since today

# Filter structured event (slog JSON)
sudo journalctl -u email-service --since "1 hour ago" -o cat | jq 'select(.event == "email_send")'

# Filter error class tertentu
sudo journalctl -u email-service --since today -o cat | \
  jq 'select(.event == "email_send" and .error_class == "smtp_auth")'

# Persistence failure events (Phase 3 Day 1 fix 2026-05-25 11:52)
sudo journalctl -u email-service --since today -o cat | \
  jq 'select(.event | startswith("email_") and endswith("_failed"))'

5.2 Event schema reference

Lihat architecture.md §5 Observability untuk schema lengkap email_send event + 6-class taxonomy + persistence failure events (email_insert_failed, email_mark_sent_failed, email_mark_failed_failed).

5.3 Phase 3 soak threshold (2026-05-25 → 2026-06-08)

Metric	Threshold	Day 1 actual	Action kalau breach
Volume / 24h	> 100	13	normal — Day 1 traffic
p50 latency	< 3s target	3.2s	tune SMTP connection pool
p95 latency	< 5s	4.0s	investigate slow recipient domain
Max latency	< 10s	4.0s	check SMTP timeout config
Error rate	< 2%	0%	investigate per error_class
Fallback events	== 0	0	check service availability
Gmail throttle (`error_class=gmail_throttle`)	== 0	0	reduce volume atau migrate provider
Service uptime	== 100%	100%	investigate systemd restart count

5.4 Alert threshold per error class

Error class	Threshold	Action
`smtp_auth` count > 0 / 5 min	Rotate `SMTP_PASSWORD` (Gmail App Password expire / leak)
`gmail_throttle` count > 5 / hour	Reduce volume atau accelerate provider migration
`gmail_blocked` count > 0 / 5 min	Investigate content (spam policy) atau sender reputation
`smtp_timeout` rate > 5% / 10 min	Cek network ke `smtp.gmail.com:587`
`smtp_config` count > 0	Service mis-configured (cek `.env.production`)
`email_*_failed` count > 0 / 5 min	DB unreachable / schema drift
`/ready` 503 > 1 min	DB unreachable atau SMTP config rusak

5.5 Daily telemetry aggregator

# Run di laptop atau VM
./scripts/email-telemetry-daily.sh --since "1 day ago"
# Output: volume per template, latency p50/p95, error_class distribution

Sumber decision deterministik untuk Phase 3 soak GO/NO-GO 2026-06-08.

6. Rollback procedure

Kalau redeploy bermasalah:

6.1 Rollback via systemd

# 1. SSH ke VM, lihat last working binary kalau ada backup
ls -la ~/kesles_merchant/merchant_email/

# Default: tidak ada backup (manual practice belum baku, TODO improvement)

6.2 Rollback via git checkout + rebuild

# Di laptop
cd ~/Macbook\ pro/development/kesles_merchant/services/email_service
git log --oneline -5                          # cari commit hash stable
git checkout <commit-hash> -- .               # checkout source
./scripts/build_email_service.sh linux-prod
scp dist/email-service enalfarid@ptikn3-vm.cluster-vps.dalang.io:~/kesles_merchant/merchant_email/

# Di VM
ssh enalfarid@ptikn3-vm.cluster-vps.dalang.io 'sudo systemctl restart email-service'

# Restore source di laptop
git checkout main -- .

6.3 Rollback Phase 4 (emergency)

~~Phase 2 cutover rollback~~ TIDAK APPLICABLE — Phase 4 DONE 2026-05-29. core_api/internal/email/ sudah dihapus. Tidak ada lagi fallback ke direct SMTP.

Rollback saat ini hanya via:

# 1. Git revert commit Phase 4 di local repo
git revert <phase4-commit-hash>

# 2. Rebuild binary core_api
./scripts/build_core_api.sh linux-prod

# 3. Upload + restart core_api via FTP workflow

Kalau service email-service mengalami bug critical, rollback binary email-service ke versi sebelumnya (lihat §6.2). Email tidak terkirim selama service down.

6.4 Yang TIDAK perlu di-rollback

Audit row di notification.email_messages — append-only, tidak ada UPDATE/DELETE pattern
HTML template di templates/*.html (post #2 embed.FS) — embedded di binary, kalau salah render = redeploy binary lama

7. Troubleshooting

Service refuse start di production

sudo systemctl status email-service
sudo journalctl -u email-service --since "5 minutes ago"

Pesan-pesan common (post G3 validateConfig 2026-05-25):

Log	Penyebab	Fix
`INTERNAL_NOTIFICATION_API_KEY is required in production`	env kosong di `.env.production`	edit env, restart
`SMTP_FROM_EMAIL is required in production`	env kosong	edit env, restart
`POSTGRES_DSN is required in production`	env kosong	edit env, restart
`postgres init failed: ... no such host`	`POSTGRES_DSN` salah host atau DB VM down	cek DB topology
`bind: address already in use`	port 8094 dipakai process lain	`sudo ss -lntp \| grep 8094`, kill atau ganti port

Endpoint `/internal/*` selalu 503 `internal_key_not_configured`

INTERNAL_NOTIFICATION_API_KEY belum di-load. Kalau service running tapi balas 503, kemungkinan .env.production typo atau systemd unit lupa EnvironmentFile= line.

# Cek env actual yang di-load systemd
sudo systemctl show email-service | grep -i environment

SMTP auth fail (`error_class=smtp_auth`)

sudo journalctl -u email-service --since "10 minutes ago" -o cat | \
  jq 'select(.event == "email_send" and .error_class == "smtp_auth")'

Common causes:

Gmail App Password expire (90-hari rotation) — generate baru + update .env.production (lihat §4.2)
Gmail account 2FA di-disable accidentally — re-enable
App Password di-revoke (Google Security alert)

Gmail throttle (`error_class=gmail_throttle`)

Gmail SMTP limit: 500 email/hari per sending account (non-Workspace) atau 2000/hari (Workspace). Symptom: error 421 4.7.0 Try again later muncul.

Fix immediate: reduce traffic (drain queue gradually). Fix long-term: migrate ke provider transactional (SendGrid / AWS SES / Postmark — lihat roadmap extraction-plan.md).

Phase 3 Day 1 fix: silent stuck queued row

Sebelum 2026-05-25 11:52, kalau store.MarkSent() / MarkFailed() / Insert() gagal (DB hiccup), email kirim sukses tapi DB row stuck di status queued selamanya tanpa visibility.

Post-fix: service emit slog warn eksplisit (email_mark_sent_failed, email_mark_failed_failed, email_insert_failed) dengan message_id + error. Operator bisa filter:

sudo journalctl -u email-service --since "1 hour ago" -o cat | \
  jq 'select(.event == "email_mark_sent_failed")'

Manual fix row stuck: UPDATE notification.email_messages SET delivery_status='sent' WHERE id='<uuid>' AND delivery_status='queued' setelah konfirmasi email actual terkirim.

8. Disaster recovery

Service stateless — semua state di Postgres. Rebuild scenario:

Re-provision VM (atau VM baru)
Setup user enalfarid + folder ~/kesles_merchant/merchant_email/
SCP binary + .env.production (dari secret vault / 1Password)
Install systemd unit dari §2.4
systemctl enable && start
Test smoke send dari §2.5
Update core_api EMAIL_SERVICE_BASE_URL kalau VM pindah host

RTO target: <30 menit (kalau credential + binary tersedia siap). RPO: 0 (Postgres replicated, notification.email_messages row tidak hilang).

Catatan Phase 4 (DONE 2026-05-29): tidak ada lagi fallback ke direct SMTP. VM downtime = email send error langsung ke caller. Mitigasi utama: Restart=on-failure systemd + graceful shutdown 15s.

9. Dev SMTP (MailPit)

Development lokal pakai MailPit (capture all outgoing email, no real send). Install via brew install mailpit. Verified 2026-05-21:

mailpit &                                          # start background
curl -sS http://localhost:8025/api/v1/messages    # API
open http://localhost:8025                         # web UI

# Env lokal
APP_ENV=development
SMTP_HOST=localhost
SMTP_PORT=1025
SMTP_USERNAME=     # kosong = no auth
SMTP_PASSWORD=     # kosong
SMTP_FROM_EMAIL=noreply@dev.kesles.local

validateConfig() no-op di APP_ENV=development jadi env minimal cukup (tidak butuh real Gmail App Password). Bisa edit .env.development di folder service, service auto-load (lihat internal/app/config.go::loadDotenv).

10. Dokumen referensi

architecture.md — desain endpoint + persistence + observability
extraction-plan.md — Phase -1 sampai Phase 4 plan
email-service-status.md — phase tracker visual
services/firebase/phase4-readiness.md — pola sister service yang ditiru
services/whatsapp/deploy-runbook.md — pola sister service
services/email_service/README.md — service-level runbook + smoke test
scripts/email-telemetry-daily.sh — daily aggregator
scripts/build_email_service.sh — build helper

Memory references:

project_email_service_vm_deployment — VM path convention
feedback_email_service_mirror — Mirror obligation EXPIRED Phase 4 done 2026-05-29
feedback_credential_handling — Operator credential rotation note

1. VM topology​

2. Initial deploy (first-time setup)​

2.1 Build binary di laptop​

2.2 Create folder + env di VM​

2.3 SCP binary​

2.4 Install systemd unit​

2.5 Smoke test post-deploy​

3. Re-deploy (update binary)​

4. Env rotation​

4.1 Rotate INTERNAL_NOTIFICATION_API_KEY (paired dengan core_api)​

4.2 Rotate SMTP_PASSWORD (Gmail App Password 90-hari)​

4.3 Rotate ke provider transactional (future)​

5. Monitoring & alerting​

5.1 Log stream​

5.2 Event schema reference​

5.3 Phase 3 soak threshold (2026-05-25 → 2026-06-08)​

5.4 Alert threshold per error class​

5.5 Daily telemetry aggregator​

6. Rollback procedure​

6.1 Rollback via systemd​

6.2 Rollback via git checkout + rebuild​

6.3 Rollback Phase 4 (emergency)​

6.4 Yang TIDAK perlu di-rollback​

7. Troubleshooting​

Service refuse start di production​

Endpoint /internal/* selalu 503 internal_key_not_configured​

SMTP auth fail (error_class=smtp_auth)​

Gmail throttle (error_class=gmail_throttle)​

Phase 3 Day 1 fix: silent stuck queued row​

8. Disaster recovery​

9. Dev SMTP (MailPit)​

10. Dokumen referensi​