fix(database): isolate web and worker connection pools - #764
Conversation
GET / was throwing KnexTimeoutError ("Timeout acquiring a connection")
in production. Root cause was the connection pool, not the database:
the single process (SvelteKit + cron scheduler + BullMQ workers) shared
one pool capped at 10, while one GET / fans out ~6 queries. A couple of
concurrent page loads, or a per-minute monitor burst overlapping a load,
exceeded 10 and queued acquires blew past the 15s timeout. Postgres
itself had 97 free slots the whole time and no leak.
Split into two pools so background work can't starve page loads:
- web pool (DATABASE_POOL_MAX, default 10) serves HTTP requests
- worker pool (DATABASE_WORKER_POOL_MAX, default 5) serves background jobs
Routing is by execution context via AsyncLocalStorage: q.createWorker
(the single chokepoint all workers/schedulers flow through) runs each
processor inside a worker-pool context, and BaseRepository.knex resolves
the pool from that context, defaulting to the web pool. This keeps shared
controllers correct whether they run in a request or a job. SQLite has no
real pool and reuses a single connection, so the split is a no-op there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughIntroduces a dual Knex connection pool architecture: ChangesDual DB Pool Routing
Sequence Diagram(s)sequenceDiagram
participant BullMQ as BullMQ Job
participant q_ts as createWorker wrapper
participant DbImpl as DbImpl.runInWorkerContext
participant poolContext as poolContext (AsyncLocalStorage)
participant BaseRepo as BaseRepository.knex getter
participant workerKnex as Worker Knex Pool
BullMQ->>q_ts: job execution
q_ts->>DbImpl: runInWorkerContext(processor)
DbImpl->>poolContext: runWithWorkerKnex(workerKnex, fn)
poolContext->>poolContext: AsyncLocalStorage.run(workerKnex, fn)
Note over poolContext: worker Knex stored in context
poolContext->>q_ts: execute processor fn
q_ts->>BaseRepo: repository method call
BaseRepo->>poolContext: getWorkerKnex()
poolContext-->>BaseRepo: workerKnex instance
BaseRepo->>workerKnex: SQL query
workerKnex-->>BaseRepo: result
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Pull request overview
This PR addresses production KnexTimeoutError: Timeout acquiring a connection errors by preventing background work (BullMQ workers/schedulers) from exhausting the connection pool used by HTTP requests. It introduces a context-aware routing mechanism so repositories transparently use a dedicated worker pool when executing inside a job, improving reliability under concurrent load.
Changes:
- Added a second Knex pool configuration (
DATABASE_WORKER_POOL_MAX) alongside the existing web/request pool (DATABASE_POOL_MAX). - Routed BullMQ job processors into a worker execution context via
AsyncLocalStorage, and madeBaseRepositoryresolve the correct Knex instance per context. - Updated v4 documentation to describe the split pools, tuning guidance, and troubleshooting.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/routes/(docs)/docs/content/v4/setup/environment-variables.md | Documents the new DATABASE_WORKER_POOL_MAX variable and clarifies DATABASE_POOL_MAX as the web pool. |
| src/routes/(docs)/docs/content/v4/setup/database-setup.md | Adds explanation and operational guidance for budgeting/tuning the two pools and updates timeout troubleshooting text. |
| src/lib/server/queues/q.ts | Wraps BullMQ processors so background jobs run inside a worker-pool context. |
| src/lib/server/db/repositories/base.ts | Switches repository access to a context-resolved Knex instance (worker vs web fallback). |
| src/lib/server/db/poolContext.ts | Introduces AsyncLocalStorage context plumbing for selecting the worker Knex instance. |
| src/lib/server/db/dbimpl.ts | Instantiates and manages a dedicated worker Knex instance and exposes runInWorkerContext. |
| src/lib/server/db/db.ts | Wires the DB singleton to use both web and worker Knex configs. |
| knexfile.ts | Splits pool configuration into web vs worker pools and exports workerKnexOb. |
| const wrapped: Processor<T, R> = | ||
| typeof processor === "function" | ||
| ? (job, token) => db.runInWorkerContext(() => Promise.resolve(processor(job, token))) | ||
| : processor; |
| // Kener runs SvelteKit requests, the cron scheduler, and the BullMQ workers in | ||
| // a single process, all sharing one Knex instance. A burst of background jobs | ||
| // could therefore exhaust the connection pool and time out user-facing page |
|
| Filename | Overview |
|---|---|
| knexfile.ts | Adds workerKnexOb export with a second pool config for background jobs; refactors pool construction into buildPool helper. Clean change; acquireConnectionTimeout is intentionally shared by spread. |
| src/lib/server/db/poolContext.ts | New file introducing AsyncLocalStorage-based pool routing between web and worker Knex instances. Well-scoped, no issues. |
| src/lib/server/db/repositories/base.ts | Converts knex from a plain field to a getter that reads from AsyncLocalStorage, falling back to the web pool. Design is correct; all repositories inherit routing automatically. |
| src/lib/server/db/dbimpl.ts | Adds workerKnex, runInWorkerContext, and fixes close() to destroy both pools. Repositories are all constructed with the web pool (correct; routing happens via getter). |
| src/lib/server/db/db.ts | Passes workerKnexOb to DbImpl constructor; trivial wiring change. |
| src/lib/server/queues/q.ts | Wraps every BullMQ processor in db.runInWorkerContext; correctly skips sandboxed (string/URL) processors. Minor: synchronous throws from processor(job, token) escape the Promise chain — see comment. |
| src/routes/(docs)/docs/content/v4/setup/database-setup.md | Documents the new dual-pool model, DATABASE_WORKER_POOL_MAX, budgeting guidance, and updated troubleshooting entry. Accurate and helpful. |
| src/routes/(docs)/docs/content/v4/setup/environment-variables.md | Adds DATABASE_WORKER_POOL_MAX row to the env-var reference table. Matches the implementation. |
Sequence Diagram
%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant SvelteKit as SvelteKit Request
participant BullMQ as BullMQ Worker (q.createWorker)
participant ALS as AsyncLocalStorage (poolContext.ts)
participant BR as BaseRepository.knex getter
participant WEB as Web Pool (DATABASE_POOL_MAX)
participant WORKER as Worker Pool (DATABASE_WORKER_POOL_MAX)
participant PG as PostgreSQL
SvelteKit->>BR: calls repository method
BR->>ALS: getWorkerKnex()
ALS-->>BR: undefined (no context set)
BR->>WEB: query via fallbackKnex
WEB->>PG: execute SQL
BullMQ->>ALS: runWithWorkerKnex(workerKnex, processor)
ALS->>BR: getWorkerKnex() returns workerKnex
BR->>WORKER: query via workerKnex
WORKER->>PG: execute SQL
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant SvelteKit as SvelteKit Request
participant BullMQ as BullMQ Worker (q.createWorker)
participant ALS as AsyncLocalStorage (poolContext.ts)
participant BR as BaseRepository.knex getter
participant WEB as Web Pool (DATABASE_POOL_MAX)
participant WORKER as Worker Pool (DATABASE_WORKER_POOL_MAX)
participant PG as PostgreSQL
SvelteKit->>BR: calls repository method
BR->>ALS: getWorkerKnex()
ALS-->>BR: undefined (no context set)
BR->>WEB: query via fallbackKnex
WEB->>PG: execute SQL
BullMQ->>ALS: runWithWorkerKnex(workerKnex, processor)
ALS->>BR: getWorkerKnex() returns workerKnex
BR->>WORKER: query via workerKnex
WORKER->>PG: execute SQL
Reviews (1): Last reviewed commit: "fix(database): isolate web and worker co..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| typeof processor === "function" | ||
| ? (job, token) => db.runInWorkerContext(() => Promise.resolve(processor(job, token))) | ||
| : processor; |
There was a problem hiding this comment.
processor(job, token) is evaluated synchronously before Promise.resolve() is called. If a processor throws synchronously (not returns a rejected promise), the arrow function body throws rather than returning a rejected Promise. AsyncLocalStorage.run() then propagates that throw synchronously through db.runInWorkerContext, so the outer wrapper function also throws synchronously instead of returning Promise.reject(...). BullMQ does try/catch around processor invocation so in practice this is handled, but the contract mismatch could be a surprise. Wrapping with an async arrow function ensures exceptions are always channeled through the Promise: async () => processor(job, token) instead of () => Promise.resolve(processor(job, token)).
| typeof processor === "function" | |
| ? (job, token) => db.runInWorkerContext(() => Promise.resolve(processor(job, token))) | |
| : processor; | |
| typeof processor === "function" | |
| ? (job, token) => db.runInWorkerContext(async () => processor(job, token)) | |
| : processor; |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/server/db/dbimpl.ts`:
- Around line 869-873: The close() method currently awaits the web pool
destruction before attempting worker pool cleanup, which means if
this.knex.destroy() rejects, the worker pool cleanup is skipped and leaves the
resource open. Refactor the close() method to use a try/finally block or
Promise.allSettled pattern to ensure both this.knex.destroy() and
this.workerKnex.destroy() (when they are different) complete execution
regardless of whether either one throws an error.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: f5d32c9c-8682-4b8c-9fe2-acdd1bc97de9
📒 Files selected for processing (8)
knexfile.tssrc/lib/server/db/db.tssrc/lib/server/db/dbimpl.tssrc/lib/server/db/poolContext.tssrc/lib/server/db/repositories/base.tssrc/lib/server/queues/q.tssrc/routes/(docs)/docs/content/v4/setup/database-setup.mdsrc/routes/(docs)/docs/content/v4/setup/environment-variables.md
| async close(): Promise<void> { | ||
| return await this.knex.destroy(); | ||
| await this.knex.destroy(); | ||
| if (this.workerKnex !== this.knex) { | ||
| await this.workerKnex.destroy(); | ||
| } |
There was a problem hiding this comment.
Ensure the worker pool is destroyed even if web pool cleanup fails.
Line 870 awaits the web pool destroy before starting worker cleanup, so a rejection skips Line 872 and can leave the new worker pool open during shutdown/test cleanup.
Proposed fix
async close(): Promise<void> {
- await this.knex.destroy();
- if (this.workerKnex !== this.knex) {
- await this.workerKnex.destroy();
- }
+ const workerKnex = this.workerKnex !== this.knex ? this.workerKnex : undefined;
+ try {
+ await this.knex.destroy();
+ } finally {
+ if (workerKnex) {
+ await workerKnex.destroy();
+ }
+ }
}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/lib/server/db/dbimpl.ts` around lines 869 - 873, The close() method
currently awaits the web pool destruction before attempting worker pool cleanup,
which means if this.knex.destroy() rejects, the worker pool cleanup is skipped
and leaves the resource open. Refactor the close() method to use a try/finally
block or Promise.allSettled pattern to ensure both this.knex.destroy() and
this.workerKnex.destroy() (when they are different) complete execution
regardless of whether either one throws an error.
Why
GET /was 500ing in prod (kener-v4-alpha) withKnexTimeoutError: Timeout acquiring a connection. Confirmed against livepg_stat_activity: the database was healthy (97 free slots, no leak, no stuck query) — the bottleneck was our own pool.One process runs SvelteKit + cron scheduler + ~7 BullMQ worker types (concurrency 5) all sharing one pool capped at 10, and a single
GET /fans out ~6 queries. Two concurrent page loads, or a per-minute monitor burst overlapping a load, push demand past 10; queued acquires then blow past the 15s acquire timeout → 500. (Checkpoint logs only show writes, which is why the DB looked "idle".)Immediate relief was already applied by bumping
DATABASE_POOL_MAXto 30 on the service. This PR is the durable fix.What
Split into two pools so background jobs can't starve page loads:
DATABASE_POOL_MAX(default 10) — HTTP requestsDATABASE_WORKER_POOL_MAX(default 5, new) — background jobsRouting is by execution context via
AsyncLocalStorage:q.createWorkeris the single chokepoint all workers/schedulers flow through — it runs each processor inside a worker-pool context.BaseRepository.knexresolves the pool from that context, defaulting to the web pool. So shared controllers stay correct whether they run in a request or a job.Budget:
replicas × (web + worker) < max_connections. Defaults total 15, safe for small managed Postgres.Verification
npm run check→ 0 errors.base.ts/poolContext.ts): web pool outside the context, worker pool inside, survivesawait, restores after.dbsingleton: construct + ping +runInWorkerContext+close()(no double-destroy).Deploy note
After this rolls out, set on the service:
DATABASE_POOL_MAX=20,DATABASE_WORKER_POOL_MAX=10. Safe if it deploys before retuning (web 30 + worker 5 = 35 < 97).🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation