Skip to content

perf(trigger): cap concurrency on background DB tasks#5231

Open
TheodoreSpeaks wants to merge 2 commits into
stagingfrom
perf/trigger-concurrency-caps
Open

perf(trigger): cap concurrency on background DB tasks#5231
TheodoreSpeaks wants to merge 2 commits into
stagingfrom
perf/trigger-concurrency-caps

Conversation

@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator

Summary

  • Cap per-task Trigger.dev concurrency on the heavy DB-bound background tasks to mitigate the Postgres pool-exhaustion incident — unbounded fan-out spawned many machines that collectively saturated the pool.
  • New apps/sim/background/concurrency-limits.ts defines env-overridable caps via the existing envNumber pattern: workflow-execution 75 (WORKFLOW_EXECUTION_CONCURRENCY_LIMIT), webhook-execution 75 (WEBHOOK_EXECUTION_CONCURRENCY_LIMIT), resume-execution 50 (RESUME_EXECUTION_CONCURRENCY_LIMIT).
  • Lower the schedule cap default 50 → 30 (SCHEDULE_EXECUTION_CONCURRENCY_LIMIT).
  • Register the three new env vars in env.ts.

This is a strict reduction in peak concurrency — the three tasks were previously unbounded.

Validated against 30d of prod run data

  • workflow/webhook/resume caps sit well above real load (typical distinct concurrency ~2–25, max ~11 workflow runs/min, ~261 webhook runs/min in worst burst).
  • schedule at 30 intentionally bites cron-boundary bursts (p95 ~30, p99 ~64 runs/min). Schedule already queues at 50; 30 adds some latency at peak by design. Note SCHEDULE_WORKFLOW_ENQUEUE_LIMIT = LIMIT × 2 drops 100→60 in tandem.

Type of Change

  • Performance / reliability

Testing

Tested manually — bun run lint (biome) clean, bun run check:api-validation:strict passes, tsc --noEmit clean on changed files. No API boundaries touched.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@greptile review

@vercel

vercel Bot commented Jun 27, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped Jun 27, 2026 1:59am

Request Review

@cursor

cursor Bot commented Jun 27, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Reduces peak background concurrency and may queue work at bursts (especially schedules at cron boundaries); caps are tunable via env but misconfiguration could add latency under load.

Overview
Adds Trigger.dev queue concurrency limits on DB-heavy background tasks so unbounded fan-out cannot saturate the Postgres pool. New concurrency-limits.ts centralizes env-overridable caps via envNumber: workflow-execution 75, webhook-execution 75, resume-execution 50 (those three tasks had no cap before).

Schedule execution default drops 50 → 30 (SCHEDULE_EXECUTION_CONCURRENCY_LIMIT in env.ts and execution-limits.ts), which also lowers SCHEDULE_WORKFLOW_ENQUEUE_LIMIT (limit × enqueue multiplier). The existing schedule concurrency test is updated to expect 30.

Reviewed by Cursor Bugbot for commit c223eef. Bugbot is set up for automated code reviews on this repo. Configure here.

@greptile-apps

greptile-apps Bot commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR caps concurrency for DB-heavy Trigger.dev background work. The main changes are:

  • Added shared env-backed limits for workflow, webhook, and resume execution tasks.
  • Applied those limits to the three Trigger task definitions.
  • Lowered the default schedule execution cap from 50 to 30.
  • Registered the new concurrency env vars in the app config.

Confidence Score: 5/5

The changed flow looks mergeable after clarifying queue identity and updating the schedule-budget test.

  • The new env-backed limits follow the existing numeric config pattern.
  • The task caps may rely on implicit queue naming, which can make runtime tuning less predictable.
  • The schedule cap change appears to leave an old test expectation out of date.

apps/sim/background/workflow-execution.ts and apps/sim/lib/workflows/schedules/execution-limits.ts

Important Files Changed

Filename Overview
apps/sim/background/concurrency-limits.ts Adds shared env-backed concurrency limit constants using the existing envNumber helper.
apps/sim/background/workflow-execution.ts Adds a queue concurrency cap to workflow execution with implicit queue identity.
apps/sim/background/webhook-execution.ts Adds a queue concurrency cap to webhook execution with implicit queue identity.
apps/sim/background/resume-execution.ts Adds a queue concurrency cap to resume execution with implicit queue identity.
apps/sim/lib/core/config/env.ts Registers the new task concurrency env vars and lowers the schedule default.
apps/sim/lib/workflows/schedules/execution-limits.ts Lowers the schedule execution fallback, which also lowers the derived enqueue budget.

Comments Outside Diff (1)

  1. apps/sim/lib/workflows/schedules/execution-limits.ts, line 17 (link)

    P2 Stale Schedule Budget Test

    This derived budget now drops from 100 to 60 because the default concurrency is 30 and the multiplier stays 2. The existing schedule tick test that creates 100 due schedules and expects all 100 to enqueue in one tick will now see only 60 enqueues unless the test overrides the config or updates the expected budget.

Reviews (1): Last reviewed commit: "perf(trigger): cap concurrency on backgr..." | Re-trigger Greptile

export const workflowExecutionTask = task({
id: 'workflow-execution',
machine: 'medium-1x',
queue: {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Implicit Queue Sharing

When queue has a concurrencyLimit but no explicit name, these new high-volume tasks depend on Trigger.dev's implicit queue identity. If nameless queues resolve to a shared default queue, workflow, webhook, resume, and existing nameless cleanup jobs can throttle each other, so a webhook burst can delay unrelated workflow or resume runs instead of applying independent per-task caps.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@greptile-apps

greptile-apps Bot commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR caps Trigger.dev background task concurrency for heavier DB-bound work. The main changes are:

  • New shared concurrency limit constants for workflow, webhook, and resume execution.
  • Queue caps added to the three corresponding Trigger.dev tasks.
  • Schedule execution default lowered from 50 to 30.
  • New concurrency env vars registered in the app env config.

Confidence Score: 5/5

This looks safe to merge after checking how the new env overrides are injected for Trigger workers.

  • The queue configuration shape matches existing Trigger.dev task patterns.
  • The new imports follow the app alias conventions.
  • The only concern is that module-level constants may keep fallback values if the Trigger worker does not see the env overrides at module load.

apps/sim/background/concurrency-limits.ts

Important Files Changed

Filename Overview
apps/sim/background/concurrency-limits.ts Adds shared env-based Trigger.dev concurrency limit constants.
apps/sim/background/workflow-execution.ts Adds a queue concurrency cap to the workflow execution task.
apps/sim/background/webhook-execution.ts Adds a queue concurrency cap to the webhook execution task.
apps/sim/background/resume-execution.ts Adds a queue concurrency cap to the resume execution task.
apps/sim/lib/core/config/env.ts Registers new concurrency env vars and lowers the schedule execution default.
apps/sim/lib/workflows/schedules/execution-limits.ts Updates the schedule execution fallback to match the new default.

Reviews (2): Last reviewed commit: "perf(trigger): cap concurrency on backgr..." | Re-trigger Greptile

Comment on lines +5 to +8
export const WORKFLOW_EXECUTION_CONCURRENCY_LIMIT = envNumber(
env.WORKFLOW_EXECUTION_CONCURRENCY_LIMIT,
75,
{ min: 1, integer: true }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Module Load Freezes Env Defaults

When these constants are evaluated before the Trigger worker has the new env vars in process.env, envNumber falls back to 75 and the override is never read again. In deployments where Trigger-specific env vars are only injected through the Trigger project/runtime settings, lowering WORKFLOW_EXECUTION_CONCURRENCY_LIMIT would be silently ignored and the task can keep running at the default cap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant