CloudWatch · eu-central-1 · /ecs/lockin-backend-task

Ten hours of controlled failure

Production backend stability report covering heap OOM crash-loop, Postgres connection-pool starvation, and the APIs and background jobs implicated in the cascade.

Window 2026-06-30 21:03 → 2026-07-01 07:03 UTC Account 225523106684 Service lockin-backend-service Generated 2026-07-01

OOM heartbeat — 44 process deaths across 10 hours (avg 14 min apart)

2026-06-30 21:28:13.669 UTC2026-07-01 07:11:39.620 UTC

Executive stats

Three failure modes stack: heap grows until V8 kills the process, ECS replaces the task, and surviving work fights over a 10-slot request-path Postgres pool.

Heap OOM crashes

122

DB pool timeout events

14m

Avg time between OOMs

HTTP 500 with pool proof

Reached heap limit

GC mark-compact failures

Pool errors by component

Component	Events	Share
Scheduler	52
ExchangeWatchdogService	22
ArchetypeStaleSweep	19
TradeMonitorCronService	7
RankRecomputeJob	6
ArchetypeService	4
ExternalPrefetchService	4
PartnerVolumeSyncJob	2
ClosedTradesLiveSyncService	1
OrphanedLivePositionsSweep	1
NowConfigService	1
PartnerActivationSyncJob	1
InsightOutcomeWorker	1
ExceptionsHandler (API)	1

Background jobs & watchdogs (schedule)

High-frequency crons

Scheduler — Nest wrapper for all @Cron jobs (52 pool hits)
ExchangeWatchdogService — setInterval every 60s (22 hits)
ArchetypeStaleSweep — every 2–3 min recovery + daily sweep (19 hits)
RankRecomputeJob — every 5 min (6 hits)
TradeMonitorCronService — every 5 min (7 hits)
ExternalPrefetchService — every 5 min (4 hits)

Flagged user APIs

GET /api/now — confirmed 500, 19.7s, pg-pool acquire timeout
GET /api/v1/subscription — 5.1s wait before 401 (pool pressure)
NowConfigService — archetype lookup pool timeout (feeds /api/now)
Onboarding / trade-feed / insights bulk — heavy by design (code path)

How it crashes

Not a single bug — a feedback loop between memory leak, task churn, and connection starvation.

1 · Heap climb

Primary: in-process CacheModule stores large OHLCV payloads from ExternalInsightDataService — see OOM root cause
Secondary: WebSocket listener leak on reconnect (Blofin/Weex/Extended)
Node hits --max-old-space-size (4096 MB in prod) → FATAL ERROR: Allocation failed - JavaScript heap out of memory
Observed cadence: ~14 minutes between deaths; 19 crashes show GC Ineffective mark-compacts

2 · ECS churn

Task killed → ALB drains → new task starts (~12 min cycle in ECS events)
Cold start replays crons, watchdog tick, archetype recovery sweeps
Each restart opens fresh DB connections while old pool state is gone

3 · Pool starvation

Request path pool = 10 connections per ECS task (POSTGRES_POOL_SIZE)
Background jobs share same pool — no worker bulkhead enabled
Errors: timeout exceeded when trying to connect (acquire) and Connection terminated due to connection timeout

4 · API pile-up

GET /api/now fans out ~6 computeAllMetrics DB loads per screen open
Onboarding refreshForConnection + archetype refresh competes on same pool
Only errors log HTTP lines — fleet mostly crash-looping, so user-visible 500s are under-counted

OOM root cause — in-memory cache (debugged)

Code review + CloudWatch correlation. OOM is not caused by Postgres pool exhaustion — pool timeouts are a downstream symptom. The process dies because the V8 heap fills with cached insight market data that is never reclaimed.

The leak mechanism

Nest CacheModule.register() default store = Keyv backed by an unbounded Map
ExternalInsightDataService writes ExternalInsightData objects: OHLCV candle arrays, OI/liquidation history, per-trade external maps
Cache keys churn on time buckets + symbols + connection IDs — e.g. insights:external:…:b{nowBucket}:…
Keyv evicts expired entries only on re-access — churning keys are never read again → entries never freed
Heap climbs until --max-old-space-size cap → V8 FATAL (documented: exit 139 / OOM every ~12–21 min)

Evidence from this window

44 heap OOM fatals — steady ~14 min cadence, not spike-driven
25 × Reached heap limit — hard cap hit
19 × Ineffective mark-compacts near heap limit — GC cannot reclaim enough
Pool errors (122) occur between OOMs while task still alive — separate failure mode
ECS replaces task after each OOM → cold start replays crons → pool pressure worsens

Fix in repo (may not be deployed)

boundedMemoryStore() in src/common/cache/bounded-cache-store.ts — caps entry count, evicts oldest on insert
Wired in app.module.ts (2000), insights.module.ts (1500), now.module.ts (2000), briefing.module.ts (1000)
Caveat: cap is entry count, not bytes — 2000 × large OHLCV payloads can still be hundreds of MB
Prod still OOM-ing every ~15 min → likely old build without fix, or byte cap still needed

Secondary heap contributor

WebSocket services (Blofin, Weex, Extended) leak one abort listener per reconnect
Documented in blofin-websocket.service.ts — same heap-OOM class bug
prod/compose.yml references blofin-ws-oom-leak branch as complementary fix
Alone this may not explain 14 min cadence; combined with cache leak it accelerates death

Cache write path (code)

Service	What gets cached	TTL	Heap risk
`ExternalInsightDataService`	Full `ExternalInsightData` — candles, fear/greed, OI, liquidations, trade maps	10 min	Critical
`ExternalInsightDataService`	Hyperliquid `candleSnapshot` arrays per symbol/interval	5 min	Critical
`ExternalInsightDataService`	Coinalyze markets / liquidation events	6 h	High
`InsightsService`	Bulk closed-trade insight results	10 min	High
`NowScoringService`	Metric computation results per user	5 min	Medium
`BriefingCacheService`	Briefing summary / full payloads	varies	Medium

Conclusion

Yes — in-memory caching is the primary OOM cause. Specifically the unbounded (or insufficiently bounded) Nest CacheModule store feeding ExternalInsightDataService OHLCV payloads. Pool starvation and API timeouts are collateral damage from the crash-restart loop, not the heap killer.

Every pool incident (122 events)

Complete log extract sorted chronologically. Scheduler rows are generic Nest cron wrapper — underlying jobs include rank recompute, trade monitor, partner sync, reports worker, etc.

Timestamp (UTC)	Component	Error type	Message
2026-06-30 21:26:48.744	ClosedTradesLiveSyncService	Connection terminated	error [ClosedTradesLiveSyncService] Closed trades live refresh failed for connection cf5b19e5-76e1-4cd9-8609-f077102180e9: Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._conn…
2026-06-30 21:27:16.251	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 21:27:16.262	TradeMonitorCronService	Connection terminated	warn [TradeMonitorCronService] [TradeMonitorCron] Failed to load distinct users: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-06-30 22:27:37.209	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-06-30 22:27:37.223	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-06-30 22:27:37.224	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 22:27:37.225	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 22:27:37.225	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 22:27:37.226	TradeMonitorCronService	Connection terminated	warn [TradeMonitorCronService] [TradeMonitorCron] Failed to load distinct users: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-06-30 22:33:45.304	ExchangeWatchdogService	Acquire timeout	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-06-30 22:33:47.181	ExchangeWatchdogService	Acquire timeout	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-06-30 22:34:26.789	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 22:35:29.482	RankRecomputeJob	Connection terminated	error [RankRecomputeJob] Rank recompute failed: Connection terminated due to connection timeout {"service":"mentor-api","stack":[null]}
2026-06-30 22:35:29.488	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 22:36:11.707	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=6e998057-a638-4a28-8928-bca056877122: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-06-30 22:42:10.215	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 22:42:10.216	TradeMonitorCronService	Connection terminated	warn [TradeMonitorCronService] [TradeMonitorCron] Failed to load distinct users: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-06-30 22:42:14.129	ArchetypeService	Acquire timeout	warn [ArchetypeService] round-trip reconstruction failed conn=cf5b19e5-76e1-4cd9-8609-f077102180e9 userId=6e998057-a638-4a28-8928-bca056877122: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-06-30 22:42:35.188	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=456773dd-311b-41da-b856-8483e53cde77: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-06-30 22:43:01.597	ArchetypeStaleSweep	Connection terminated	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=f61a8166-7f7c-497c-a797-e5d63a1f1086: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-06-30 22:57:09.988	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 22:57:10.187	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-06-30 23:09:12.228	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=456773dd-311b-41da-b856-8483e53cde77: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-06-30 23:12:11.359	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 23:12:11.362	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 23:12:11.366	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 23:12:11.367	TradeMonitorCronService	Connection terminated	warn [TradeMonitorCronService] [TradeMonitorCron] Failed to load distinct users: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-06-30 23:12:11.367	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 23:12:28.344	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-06-30 23:14:46.114	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-06-30 23:14:46.116	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-06-30 23:14:46.122	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 23:14:46.123	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 23:14:46.124	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 23:24:16.635	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=456773dd-311b-41da-b856-8483e53cde77: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-06-30 23:28:11.316	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-06-30 23:28:47.444	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 00:00:10.243	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 00:00:10.243	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 00:00:10.243	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 00:00:10.244	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 00:00:10.244	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 00:00:10.244	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 00:00:10.245	PartnerVolumeSyncJob	Connection terminated	error [PartnerVolumeSyncJob] Partner volume sync failed: Connection terminated due to connection timeout {"service":"mentor-api","stack":[null]}
2026-07-01 00:00:10.245	OrphanedLivePositionsSweep	Connection terminated	warn [OrphanedLivePositionsSweepService] orphaned open_positions_live sweep failed: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 00:01:14.766	NowConfigService	Connection terminated	warn [NowConfigService] NowConfigService.resolve(6d371d5d-efff-41ac-a1c8-da3675d84fc3) — archetype lookup failed, using default. Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 00:08:11.576	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 00:35:14.977	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 00:35:15.251	RankRecomputeJob	Connection terminated	error [RankRecomputeJob] Rank recompute failed: Connection terminated due to connection timeout {"service":"mentor-api","stack":[null]}
2026-07-01 00:35:35.156	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 00:37:02.986	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 00:59:46.329	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 01:01:14.561	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 01:13:11.527	ArchetypeStaleSweep	Connection terminated	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=456773dd-311b-41da-b856-8483e53cde77: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 01:13:11.528	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 01:13:20.084	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 01:13:20.085	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 01:13:20.598	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=6e998057-a638-4a28-8928-bca056877122: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 01:13:45.285	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=f61a8166-7f7c-497c-a797-e5d63a1f1086: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 01:15:23.847	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 01:15:23.848	ExternalPrefetchService	Connection terminated	warn [ExternalPrefetchWorker] Prefetch cron failed: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 01:15:23.848	RankRecomputeJob	Connection terminated	error [RankRecomputeJob] Rank recompute failed: Connection terminated due to connection timeout {"service":"mentor-api","stack":[null]}
2026-07-01 01:15:23.849	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 01:15:23.850	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 01:15:23.850	PartnerVolumeSyncJob	Connection terminated	error [PartnerVolumeSyncJob] Partner volume sync failed: Connection terminated due to connection timeout {"service":"mentor-api","stack":[null]}
2026-07-01 01:15:23.851	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 01:15:34.064	ArchetypeService	Acquire timeout	warn [ArchetypeService] round-trip reconstruction failed conn=cf5b19e5-76e1-4cd9-8609-f077102180e9 userId=6e998057-a638-4a28-8928-bca056877122: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 01:17:08.422	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 01:17:08.423	TradeMonitorCronService	Connection terminated	warn [TradeMonitorCronService] [TradeMonitorCron] Failed to load distinct users: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 01:20:12.093	PartnerActivationSyncJob	Connection terminated	warn [PartnerActivationSyncJob] Activation sync failed for 11bd3c66-bac4-4b53-8a96-9d349e13cbb2: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 01:21:10.725	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 01:21:28.067	ArchetypeService	Acquire timeout	warn [ArchetypeService] round-trip reconstruction failed conn=04c7824b-1b18-4408-8fd0-821f52674e80 userId=456773dd-311b-41da-b856-8483e53cde77: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 01:21:36.948	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=456773dd-311b-41da-b856-8483e53cde77: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 01:35:12.643	RankRecomputeJob	Connection terminated	error [RankRecomputeJob] Rank recompute failed: Connection terminated due to connection timeout {"service":"mentor-api","stack":[null]}
2026-07-01 01:36:20.299	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 01:36:30.687	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 01:36:30.699	ArchetypeService	Acquire timeout	warn [ArchetypeService] round-trip reconstruction failed conn=04c7824b-1b18-4408-8fd0-821f52674e80 userId=456773dd-311b-41da-b856-8483e53cde77: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 02:02:08.628	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 02:02:08.629	TradeMonitorCronService	Connection terminated	warn [TradeMonitorCronService] [TradeMonitorCron] Failed to load distinct users: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 02:07:09.275	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 02:07:09.279	TradeMonitorCronService	Connection terminated	warn [TradeMonitorCronService] [TradeMonitorCron] Failed to load distinct users: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 02:07:09.280	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 02:07:09.280	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 02:07:09.281	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 02:07:09.306	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 02:07:09.307	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 02:26:09.305	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 02:37:25.854	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 02:50:19.708	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 02:54:11.539	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=456773dd-311b-41da-b856-8483e53cde77: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 03:06:06.557	InsightOutcomeWorker	Acquire timeout	warn [InsightOutcomeWorker] score outcome failed (496f6663-0842-49f9-80b6-da9513162867-trade:RB-WR-004 u=5e2de41a-4c8e-4f26-bdaf-26942496edd2): timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 03:09:15.005	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=456773dd-311b-41da-b856-8483e53cde77: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 03:09:35.318	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=6e998057-a638-4a28-8928-bca056877122: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 03:10:02.064	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=f61a8166-7f7c-497c-a797-e5d63a1f1086: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 03:36:10.168	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=5e2de41a-4c8e-4f26-bdaf-26942496edd2: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 03:37:19.709	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 03:39:14.730	ArchetypeStaleSweep	Connection terminated	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=456773dd-311b-41da-b856-8483e53cde77: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 03:39:14.755	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 03:53:47.977	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 03:54:09.394	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 03:54:09.587	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 04:20:01.362	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 04:23:12.144	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 04:45:14.456	RankRecomputeJob	Connection terminated	error [RankRecomputeJob] Rank recompute failed: Connection terminated due to connection timeout {"service":"mentor-api","stack":[null]}
2026-07-01 04:52:36.282	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=5e2de41a-4c8e-4f26-bdaf-26942496edd2: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 04:53:00.747	ArchetypeStaleSweep	Connection terminated	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=f61a8166-7f7c-497c-a797-e5d63a1f1086: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 04:53:44.730	ExchangeWatchdogService	Connection terminated	warn [ExchangeWatchdogService] Exchange watchdog tick failed: Error: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 04:54:31.429	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 04:54:31.432	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 04:57:12.659	ArchetypeStaleSweep	Acquire timeout	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=6e998057-a638-4a28-8928-bca056877122: timeout exceeded when trying to connect {"service":"mentor-api"}
2026-07-01 05:16:08.256	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 05:17:08.359	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 06:10:13.444	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 06:10:15.505	ArchetypeStaleSweep	Connection terminated	warn [ArchetypeStaleSweep] nodata-recovery refresh failed userId=37c46997-fb77-4ccc-bfeb-f846bf71dc66: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 06:10:15.531	RankRecomputeJob	Connection terminated	error [RankRecomputeJob] Rank recompute failed: Connection terminated due to connection timeout {"service":"mentor-api","stack":[null]}
2026-07-01 06:11:16.308	ExternalPrefetchService	Connection terminated	warn [ExternalPrefetchService] refreshPrefetch failed symbol=SOL: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 06:11:16.326	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 06:25:14.424	ExternalPrefetchService	Connection terminated	warn [ExternalPrefetchService] refreshPrefetch failed symbol=XYZ: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 06:33:05.865	ExternalPrefetchService	Connection terminated	warn [ExternalPrefetchService] refreshPrefetch failed symbol=ENA: Connection terminated due to connection timeout {"service":"mentor-api"}
2026-07-01 06:33:18.739	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 06:33:18.740	Scheduler	Connection terminated	error [Scheduler] Connection terminated due to connection timeout {"service":"mentor-api","stack":["Error: Connection terminated due to connection timeout\n at Client._connectionCallback (/app/node_modules/pg-pool/index.js:262:17)\n at Connection.<anonymous> (/app/node_modu…
2026-07-01 06:34:16.331	ExceptionsHandler (API)	Acquire timeout	error [6652e0d2-c650-4b5e-ad41-394613a9a086][ExceptionsHandler] timeout exceeded when trying to connect {"service":"mentor-api","stack":["Error: timeout exceeded when trying to connect\n at Timeout._onTimeout (/app/node_modules/pg-pool/index.js:216:27)\n at listOnTimeout (n…

Every OOM crash (44 events)

Each row is a process-level V8 fatal. Task restarts immediately after; no graceful drain.

Timestamp (UTC)	Variant	Message
2026-06-30 21:28:13.669	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-06-30 21:57:06.360	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-06-30 22:10:52.503	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-06-30 22:44:02.378	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-06-30 22:59:23.107	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-06-30 23:16:09.608	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-06-30 23:29:39.917	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-06-30 23:43:56.962	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-06-30 23:58:02.532	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 00:06:40.604	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 00:10:09.884	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 00:13:36.449	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 00:19:05.367	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 00:19:10.866	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 00:36:30.971	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 00:44:06.780	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 00:49:11.568	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 01:01:52.915	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 01:17:13.594	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 01:21:48.602	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 01:36:38.941	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 01:52:56.165	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 02:04:13.164	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 02:08:55.110	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 02:20:12.315	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 02:38:12.060	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 02:51:58.803	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 03:10:19.831	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 03:20:46.053	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 03:40:11.587	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 03:56:43.853	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 04:07:37.075	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 04:24:36.319	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 04:37:44.900	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 04:56:22.724	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 05:07:28.928	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 05:24:03.287	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 05:38:19.445	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 05:55:18.170	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 06:12:05.698	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 06:26:24.685	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 06:44:22.562	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 06:56:15.475	Heap limit reached	FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2026-07-01 07:11:39.620	Mark-compact failure	FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Flagged HTTP endpoints

HTTP logger only emits 4xx/5xx. Scanner/probe traffic (unrelated vulnerability scans) excluded. Only Lockin app endpoints shown.

Timestamp	Method	Path	Status	Duration	Notes
2026-07-01 06:34:16.331	GET	/api/now	500	19,743ms	Pool acquire timeout — only confirmed 500
2026-07-01 06:23:47.388	GET	/api/v1/subscription	401	5,140ms	Slow — likely waited on pool before auth fail
2026-07-01 06:16:26.260	POST	/api/exchanges/hyperliquid/link-direct	409	21ms	Conflict — already linked
2026-07-01 06:16:04.516	POST	/api/exchanges/hyperliquid/link-direct	409	12ms	Conflict — already linked
2026-07-01 06:11:52.719	POST	/api/insights/session/end	401	1ms	Unauthenticated
2026-07-01 06:11:37.415	POST	/api/notifications/tokens	401	1ms	Unauthenticated
2026-07-01 06:04:08.892	GET	/api/exchanges/trades	404	7ms	Route not found
2026-07-01 06:01:49.099	GET	/api/exchanges/connections	401	0ms	Unauthenticated
2026-07-01 05:55:30.773	GET	/api/rewards/summary	403	51ms	Forbidden

Heavy APIs (code-identified, pool risk)

Endpoint	Why it stresses the pool
GET /api/now	5-metric score; ~6 parallel `computeAllMetrics` fan-outs per screen
GET /api/now/:metric	Per-metric scoring reload
GET /api/now/:metric/trend	Trend + metrics DB reads
POST /api/onboarding/exchange/connect	Import + `closedTradesLive.refreshForConnection` + archetype refresh
GET /api/trade-feed/open-fast	Connection-bundle fan-out for open trades
GET /api/trade-feed/closed-fast	Closed-trade bundle load
POST /api/insights/trades/closed/bulk	Bulk PRD insight compute per trade page
GET /api/insights/curriculum	Ranked insight curriculum query
POST /api/archetype/refresh	Force `refreshNow` + fp-svc + DB
GET /api/reports/*	Daily/weekly/monthly report computation

Recommended fixes

Ordered by impact. Heap fix stops the crash loop; pool isolation stops the cascade; infra alignment stops false kills.

P0 — stop the bleed

Deploy bounded in-process cache (OOM root cause)

Ship boundedMemoryStore() for all CacheModule.register() calls. Stops unbounded Keyv Map growth from ExternalInsightDataService OHLCV payloads. Keys churn on time buckets; expired entries never re-accessed → never freed → heap OOM every ~14 min. Verify this is actually deployed to ECS — prod still crash-looping suggests old image or insufficient byte cap.

P0 — stop the bleed

Add byte-weighted cache eviction (follow-up)

boundedMemoryStore(2000) caps entry count, not size. One ExternalInsightData object can hold thousands of candles across symbols. Consider max-bytes eviction or moving OHLCV to Redis with TTL.

P0 — stop the bleed

Ship WebSocket listener leak fixes

Blofin/Weex/Extended WS services: detach abort listeners on normal completion. Secondary heap contributor — same OOM class documented in blofin-websocket.service.ts.

P1 — pool isolation

Enable worker DB bulkhead

Set WORKER_DB_POOL_ENABLED=true so BullMQ processors and crons use WORKER_POOL_SIZE (5), not the 10-slot request path pool.

P1 — pool isolation

Move crons off request pool

Scheduler (52 hits), ExchangeWatchdog (22), ArchetypeStaleSweep (19) must not share POSTGRES_POOL_SIZE with HTTP. Route through worker DataSource or dedicated read pool.

P2 — API hardening

Single-flight already on Now — extend coverage

computeAllMetrics dedup exists; ensure all /api/now/* routes share one flight. Cap concurrent metric detail fetches from Flutter.

P2 — API hardening

Decouple onboarding fingerprint from closedTradesLive build

Feed fp-svc from in-memory round-trip reconstruction — skip heavy refreshForConnection on card hot path (onboarding-scale-architecture.md).

P2 — API hardening

Throttle background sweeps during saturation

Skip ArchetypeStaleSweep / ExternalPrefetch when admission control is saturated or pool probe fails.

P3 — infra

RDS Proxy cutover

Removes Aurora Serverless scaling lag that amplifies acquire timeouts. Allows tuning pool sizes per CONNECTION_BUDGET.md.

P3 — infra

ALB health → /api/health/live

Stop killing busy-but-alive tasks on GET /api 5s timeout. Prevents restart cascade on top of OOM.

P3 — observability

Log successful HTTP at sample rate

Current logger only emits 4xx/5xx — under-reports API load during incidents. Add sampled info-level HTTP for volume attribution.