WhatsApp Service — VM Deploy Runbook

Status: 🟢 Active (mirror pola firebase + email) Owner: Backend Lead Latest: 2026-05-25 — pattern systemd unit + binary statik, folder layout mirror firebase-service (per memory project_firebase_service_vm_deployment).

Dokumen terkait:

architecture.md — desain endpoint + persistence + observability
webhook-meta-setup.md — registrasi webhook di Meta dashboard
verification-checklist.md — production readiness checklist
services/firebase/phase4-readiness.md — pola sister service yang ditiru
services/whatsapp_service/README.md — service-level runbook

1. VM topology

Item	Value
Host	`ptikn3-vm.cluster-vps.dalang.io`
User / Group	`enalfarid` / `enalfarid`
Folder root	`/home/enalfarid/kesles_merchant/merchant_whatsapp/`
Binary nama	`whatsapp-service` (Linux ELF amd64 stripped)
Env file	`.env.production` (chmod 644, gitignored)
Systemd unit	`/etc/systemd/system/whatsapp-service.service`
Listen port	`127.0.0.1:8091` (loopback internal, dipakai core_api)
Logs	`sudo journalctl -u whatsapp-service [-f] [--since="1 hour ago"]`
Reverse proxy	Caddy / nginx (public webhook callback only, lihat §6)

Pattern folder + systemd unit identik dengan merchant_firebase/ + merchant_email/ supaya operator tidak belajar layout baru per service.

2. Initial deploy (first-time setup)

2.1 Build binary di laptop

cd ~/Macbook\ pro/development/kesles_merchant/services/whatsapp_service

# Cross-compile Linux amd64 stripped
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
  go build \
    -trimpath \
    -ldflags="-s -w" \
    -o dist/whatsapp-service \
    ./cmd/server

# Verifikasi ukuran (~8-12 MB)
ls -lh dist/whatsapp-service
file dist/whatsapp-service  # → ELF 64-bit LSB executable, x86-64, statically linked

2.2 Create folder + env di VM

ssh enalfarid@ptikn3-vm.cluster-vps.dalang.io
mkdir -p ~/kesles_merchant/merchant_whatsapp/
cd ~/kesles_merchant/merchant_whatsapp/

# Copy template lalu isi credential
cp ~/kesles_merchant/services/whatsapp_service/.env.example .env.production
chmod 644 .env.production
nano .env.production   # → set INTERNAL_NOTIFICATION_API_KEY, WHATSAPP_API_KEY, WHATSAPP_API_SECRET, WHATSAPP_VERIFY_TOKEN, POSTGRES_DSN

Env wajib di production (kalau salah satu kosong, service refuse start via validateConfig):

Env	Sumber
`APP_ENV=production`	static
`INTERNAL_NOTIFICATION_API_KEY`	rotate sync dengan caller `WHATSAPP_SERVICE_API_KEY` di core_api
`WHATSAPP_API_KEY`	Meta WhatsApp Business API token (System User permanent token)
`WHATSAPP_API_SECRET`	Meta App Secret (untuk HMAC webhook verify)
`WHATSAPP_VERIFY_TOKEN`	random string, daftar di Meta webhook config + match dengan ini
`WHATSAPP_PHONE_NUMBER_ID`	Meta WABA phone number ID
`WHATSAPP_PROVIDER=meta-cloud-api`	static
`WHATSAPP_BASE_URL=https://graph.facebook.com/v23.0`	static
`WHATSAPP_ALLOWED_TEMPLATES`	comma-separated whitelist (`merchant_otp_code,merchant_payment_receipt,...`)
`WHATSAPP_TEMPLATE_NAME=merchant_otp_code`	OTP default template
`WHATSAPP_TEMPLATE_LANGUAGE=id`	static
`POSTGRES_DSN`	`postgres://kesles:...@<DB_HOST>:5432/db_kesles_merchant?sslmode=disable`

2.3 SCP binary

# Dari laptop
scp dist/whatsapp-service enalfarid@ptikn3-vm.cluster-vps.dalang.io:~/kesles_merchant/merchant_whatsapp/

# Verifikasi di VM
ssh enalfarid@ptikn3-vm.cluster-vps.dalang.io 'ls -lh ~/kesles_merchant/merchant_whatsapp/whatsapp-service'

2.4 Install systemd unit

Buat /etc/systemd/system/whatsapp-service.service:

[Unit]
Description=Kesles Merchant - WhatsApp Service
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=simple
User=enalfarid
Group=enalfarid
WorkingDirectory=/home/enalfarid/kesles_merchant/merchant_whatsapp
EnvironmentFile=/home/enalfarid/kesles_merchant/merchant_whatsapp/.env.production
ExecStart=/home/enalfarid/kesles_merchant/merchant_whatsapp/whatsapp-service
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal
SyslogIdentifier=whatsapp-service

# Hardening (selaras firebase + email)
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=read-only
PrivateTmp=true
ReadWritePaths=/home/enalfarid/kesles_merchant/merchant_whatsapp

[Install]
WantedBy=multi-user.target

Enable + start:

sudo systemctl daemon-reload
sudo systemctl enable whatsapp-service
sudo systemctl start whatsapp-service
sudo systemctl status whatsapp-service     # Expected: active (running)

2.5 Smoke test post-deploy

# 1. Liveness (selalu 200 saat process up)
curl -s http://127.0.0.1:8091/health | jq
# Expected: {"service":"whatsapp-service","status":"ok"}

# 2. Readiness (DB ping + provider config)
curl -s http://127.0.0.1:8091/ready | jq
# Expected: {"service":"whatsapp-service","status":"ready","checks":{"postgres":"ok","provider":"ok"}}

# 3. Internal auth wall (tanpa header → 401)
curl -s -X POST http://127.0.0.1:8091/internal/whatsapp/otp \
  -H "Content-Type: application/json" \
  -d '{"phone":"628000000000","code":"000000"}'
# Expected: {"error":"unauthorized"}

# 4. Internal endpoint dengan header (test ke nomor sandbox dulu, JANGAN nomor live)
KEY=$(grep "^INTERNAL_NOTIFICATION_API_KEY=" .env.production | cut -d= -f2)
curl -s -X POST http://127.0.0.1:8091/internal/whatsapp/otp \
  -H "Content-Type: application/json" \
  -H "X-Internal-API-Key: $KEY" \
  -d '{"phone":"<sandbox-phone>","code":"123456"}'
# Expected: {"status":"queued","message_id":"<uuid>","provider_message_id":"wamid..."}

3. Re-deploy (update binary)

# 1. Build di laptop
cd ~/Macbook\ pro/development/kesles_merchant/services/whatsapp_service
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
  go build -trimpath -ldflags="-s -w" -o dist/whatsapp-service ./cmd/server

# 2. SCP (overwrite binary)
scp dist/whatsapp-service enalfarid@ptikn3-vm.cluster-vps.dalang.io:~/kesles_merchant/merchant_whatsapp/

# 3. Restart service (zero-downtime kurang dari 2 detik)
ssh enalfarid@ptikn3-vm.cluster-vps.dalang.io 'sudo systemctl restart whatsapp-service'

# 4. Verify (jangan skip — kalau startup gagal validateConfig, systemd restart loop)
ssh enalfarid@ptikn3-vm.cluster-vps.dalang.io \
  'sudo systemctl status whatsapp-service && curl -s http://127.0.0.1:8091/ready | jq'

Selama restart, ada window ~2 detik service tidak respond. Caller core_api sudah punya retry policy (lihat internal/whatsapp/service_sender.go), jadi in-flight request bakal di-retry, tidak hilang.

4. Env rotation

4.1 Rotate `INTERNAL_NOTIFICATION_API_KEY` (paired dengan core_api)

Service ini + caller core_api harus rotate sekaligus supaya tidak ada window auth mismatch.

# 1. Generate new key (32 char minimum)
NEW_KEY=$(openssl rand -base64 48 | tr -d '/+=' | head -c 48)

# 2. Di VM whatsapp: update .env.production
ssh enalfarid@ptikn3-vm.cluster-vps.dalang.io
nano ~/kesles_merchant/merchant_whatsapp/.env.production
# → INTERNAL_NOTIFICATION_API_KEY=<NEW_KEY>

# 3. Di VM core_api: update .env.production
nano ~/kesles_merchant/merchant_core_api/.env.production
# → WHATSAPP_SERVICE_API_KEY=<NEW_KEY> (same value)

# 4. Restart kedua service hampir bersamaan
sudo systemctl restart whatsapp-service && sudo systemctl restart merchant-core-api

# 5. Verify dengan trigger OTP dari mobile / dashboard

Window mismatch ~5 detik antara dua restart. Mobile yang request OTP saat window itu bakal dapat 401 → minta user retry.

4.2 Rotate `WHATSAPP_API_KEY` (Meta token)

Meta System User token tidak expire kalau di-generate dengan "Never expires" checkbox. Tapi kalau di-rotate manual (mis. token leak):

# 1. Generate token baru di Meta Business Manager → System Users → Generate token
# 2. Update .env.production di VM
nano ~/kesles_merchant/merchant_whatsapp/.env.production
# → WHATSAPP_API_KEY=<NEW_TOKEN>

# 3. Restart
sudo systemctl restart whatsapp-service

# 4. Smoke test send template

4.3 Rotate `WHATSAPP_API_SECRET` (Meta App Secret)

App Secret di-rotate jarang (hanya saat App Review baru atau secret leak). Sequence:

# 1. Reset di Meta Business → My Apps → <App> → Settings → Basic → Show App Secret → Reset
# 2. CATAT secret lama (sebagai fallback kalau ada in-flight webhook)
# 3. Update .env.production
# 4. Restart service
# 5. Test webhook handshake dari Meta UI (Test → Send sample message)

Selama rotation window (~10 detik), webhook callback dari Meta dengan signature lama bakal di-reject 401 oleh validateMetaSignature — Meta auto-retry 3× dengan backoff, jadi data tidak hilang.

5. Monitoring & alerting

5.1 Log stream

# Real-time tail
sudo journalctl -u whatsapp-service -f

# Hari ini
sudo journalctl -u whatsapp-service --since today

# Filter structured event (slog JSON)
sudo journalctl -u whatsapp-service --since "1 hour ago" -o cat | jq 'select(.event == "whatsapp_send")'

# Filter error class tertentu
sudo journalctl -u whatsapp-service --since today -o cat | \
  jq 'select(.event == "whatsapp_send" and .error_class == "meta_auth")'

5.2 Event schema reference

Lihat architecture.md §6 Observability untuk schema lengkap whatsapp_send + whatsapp_webhook event + error class taxonomy. Aggregator (Loki / ELK) filter by JSON field, bukan regex stdout.

5.3 Alerting threshold

Event	Threshold	Action
`error_class=meta_auth` count > 0 / 5 min	Investigate immediately	Token rotate atau revoke event
`error_class=meta_rate_limit` count > 0 / 5 min	Quota review	Escalate ke Meta Business Support
`error_class=meta_template_invalid` count > 0	Investigate template config	Compare cfg `WHATSAPP_ALLOWED_TEMPLATES` vs Meta Approved
`error_class=meta_5xx` rate > 5% / 10 min	Monitoring	Biasanya self-recover, tidak action
`event=whatsapp_webhook` `status=rejected` count > 0	Signature mismatch	Cek `WHATSAPP_API_SECRET` sync dengan Meta UI
`/ready` 503 > 1 min	DB unreachable	Cek `POSTGRES_DSN` + DB host network

6. Public webhook exposure (reverse proxy)

whatsapp-service listen di 127.0.0.1:8091 (loopback only). Webhook callback dari Meta perlu reachable lewat HTTPS publik di domain wa.kesles.com/webhooks/whatsapp/status (lihat webhook-meta-setup.md).

Contoh snippet Caddy:

wa.kesles.com {
  # Hanya expose path webhook — endpoint /internal/* DI-BLOCK PUBLIC
  @webhook path /webhooks/whatsapp/status
  reverse_proxy @webhook 127.0.0.1:8091

  # Semua path lain → 404 (defense in depth — middleware service sudah
  # fail-closed, tapi proxy layer jadi guard tambahan)
  respond 404
}

JANGAN expose /internal/* ke public domain. Service punya fail-closed auth middleware via INTERNAL_NOTIFICATION_API_KEY, tapi defense-in-depth: proxy layer tolak path internal sebelum sampai service.

7. Rollback procedure

Kalau redeploy bermasalah:

# 1. SSH ke VM, lihat binary backup (kalau ada — manual practice belum baku)
ls -la ~/kesles_merchant/merchant_whatsapp/

# 2. Rollback ke binary commit sebelumnya
cd ~/Macbook\ pro/development/kesles_merchant/services/whatsapp_service
git log --oneline -5                    # cari commit hash terakhir yang stable
git checkout <commit-hash> -- .         # checkout source ke commit
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags="-s -w" -o dist/whatsapp-service ./cmd/server
scp dist/whatsapp-service enalfarid@ptikn3-vm.cluster-vps.dalang.io:~/kesles_merchant/merchant_whatsapp/

# 3. Restart
ssh enalfarid@ptikn3-vm.cluster-vps.dalang.io 'sudo systemctl restart whatsapp-service'
git checkout main -- .                  # restore source di laptop

Future improvement (TODO): bikin scripts/build_whatsapp_service.sh yang otomatis preserve last binary sebagai whatsapp-service.prev sebelum overwrite, supaya rollback tinggal mv whatsapp-service.prev whatsapp-service && systemctl restart. Pola sama dengan scripts/build_email_service.sh.

8. Troubleshooting

Service refuse start di production

sudo systemctl status whatsapp-service
sudo journalctl -u whatsapp-service --since "5 minutes ago"

Pesan-pesan common:

Log	Penyebab	Fix
`INTERNAL_NOTIFICATION_API_KEY is required in production`	env kosong di `.env.production`	edit env, restart
`WHATSAPP_API_SECRET is required in production`	env kosong	edit env, restart
`WHATSAPP_VERIFY_TOKEN is required in production`	env kosong	edit env, restart
`postgres init failed: ... no such host`	`POSTGRES_DSN` salah host atau DB VM down	cek DB topology
`bind: address already in use`	port 8091 dipakai process lain	`sudo ss -lntp \| grep 8091`, kill atau ganti port

Endpoint `/internal/*` selalu 503 `internal_key_not_configured`

INTERNAL_NOTIFICATION_API_KEY belum di-load. Kalau service running tapi balas 503, kemungkinan .env.production typo atau systemd unit lupa EnvironmentFile= line.

# Cek env actual yang di-load systemd
sudo systemctl show whatsapp-service | grep -i environment

Webhook signature rejected

sudo journalctl -u whatsapp-service --since "10 minutes ago" -o cat | \
  jq 'select(.event == "whatsapp_webhook" and .status == "rejected")'

Common causes:

WHATSAPP_API_SECRET di service ≠ App Secret di Meta UI (sync mismatch)
Reverse proxy mengubah body (Caddy/nginx body buffering atau gzip decompress — HMAC butuh raw body utuh)
Meta App Secret di-reset tapi service belum di-restart

Template send 422 `template_not_allowed`

Template name belum di whitelist env WHATSAPP_ALLOWED_TEMPLATES. Tambah template ke list (comma-separated), restart service. Catat: template juga harus Approved di Meta WA Manager — whitelist di service adalah extra guard, bukan pengganti approval Meta.

9. Disaster recovery

Service stateless — semua state di Postgres. Rebuild scenario:

Re-provision VM (atau VM baru)
Setup user enalfarid + folder ~/kesles_merchant/merchant_whatsapp/
SCP binary + .env.production (dari secret vault / 1Password)
Install systemd unit dari §2.4
systemctl enable && start
Update reverse proxy / DNS untuk arahkan ke VM baru
Test webhook Meta dengan Send sample message

RTO target: <30 menit (kalau credential + binary tersedia siap). RPO: 0 (Postgres replicated, message audit row tidak hilang).

10. Dokumen referensi

architecture.md — desain endpoint + observability
webhook-meta-setup.md — registrasi webhook Meta
verification-checklist.md — pre-launch checklist
whatsapp-dual-dispatch.md — dispatcher pattern di core_api
services/firebase/phase4-readiness.md — pola sister service yang ditiru

1. VM topology​

2. Initial deploy (first-time setup)​

2.1 Build binary di laptop​

2.2 Create folder + env di VM​

2.3 SCP binary​

2.4 Install systemd unit​

2.5 Smoke test post-deploy​

3. Re-deploy (update binary)​

4. Env rotation​

4.1 Rotate INTERNAL_NOTIFICATION_API_KEY (paired dengan core_api)​

4.2 Rotate WHATSAPP_API_KEY (Meta token)​

4.3 Rotate WHATSAPP_API_SECRET (Meta App Secret)​

5. Monitoring & alerting​

5.1 Log stream​

5.2 Event schema reference​

5.3 Alerting threshold​

6. Public webhook exposure (reverse proxy)​

7. Rollback procedure​

8. Troubleshooting​

Service refuse start di production​

Endpoint /internal/* selalu 503 internal_key_not_configured​

Webhook signature rejected​

Template send 422 template_not_allowed​

9. Disaster recovery​

10. Dokumen referensi​