Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
ef9bea8
docs(site): add canonical stem vs bullmq comparison
kingwill101 Feb 24, 2026
6321d54
feat(stem): close bullmq parity gaps across events, control, and stores
kingwill101 Feb 24, 2026
ecb4f22
fix(brokers): allow broadcast-only subscriptions for queue events
kingwill101 Feb 24, 2026
b7684eb
fix(stem): address retry timing, batch idempotency, and flaky tests
kingwill101 Feb 24, 2026
5559a24
fix(canvas): preserve stored batch task id order
kingwill101 Feb 24, 2026
937f980
feat(clock): add shared stem clock abstraction
kingwill101 Feb 24, 2026
ab50691
refactor(stem): use shared clock across runtime modules
kingwill101 Feb 24, 2026
599c438
refactor(adapters): route backend time reads through stem clock
kingwill101 Feb 24, 2026
8e8dc15
refactor(cli,dashboard): align timestamps with stem clock
kingwill101 Feb 24, 2026
90c9257
docs: address review nits across parity and event docs
kingwill101 Feb 24, 2026
da08861
feat(core): tighten event semantics and UTC clock behavior
kingwill101 Feb 24, 2026
af15c63
test(adapters): harden revoke and queue-event contract coverage
kingwill101 Feb 24, 2026
4f4caf3
fix(worker): improve control semantics and timing robustness
kingwill101 Feb 24, 2026
e2f75b0
feat(canvas): tighten batch inspection and submission idempotency
kingwill101 Feb 24, 2026
477ed1a
fix(redis): preserve broadcast consumer groups and cleanup disposal
kingwill101 Feb 24, 2026
0f44afd
fix(signals): freeze control command timestamps
kingwill101 Feb 24, 2026
3de91d1
fix(redis): remove unnecessary non-null assertions in consume loop
kingwill101 Feb 24, 2026
39078ca
fix(redis): retain broadcast fan-out for queue subscriptions
kingwill101 Feb 24, 2026
603e805
fix(redis): repair broadcast NOGROUP recovery path
kingwill101 Feb 24, 2026
1ebe7e9
fix(redis): guard delivery emission after stream cancellation
kingwill101 Feb 24, 2026
23139ab
fix(redis): harden claim timer delivery on closed streams
kingwill101 Feb 24, 2026
99c3dc3
test(adapters): avoid auto-cancel race in broadcast fanout contract
kingwill101 Feb 24, 2026
1e0d73a
test(adapters): pre-arm broadcast listeners before fanout publish
kingwill101 Feb 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .site/docs/brokers/sqlite.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ sidebar_position: 2
slug: /brokers/sqlite
---

Stem ships a SQLite adapter in `stem_sqlite` that implements both the broker
and result backend contracts. It is designed for local development, demo
Stem ships a SQLite adapter in `stem_sqlite` that implements broker, result
backend, and revoke store contracts. It is designed for local development, demo
environments, and single-node deployments that want a zero-infra dependency.

## When to use SQLite
Expand Down Expand Up @@ -41,6 +41,12 @@ dependencies:

```

## Quick start (revoke store)

```dart title="persistence.dart" file=<rootDir>/../packages/stem/example/docs_snippets/lib/persistence.dart#persistence-revoke-store-sqlite

```

## Configuration knobs

SQLite adapters expose the same tuning hooks as other brokers/backends:
Expand Down
62 changes: 62 additions & 0 deletions .site/docs/comparisons/stem-vs-bullmq.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
---
title: Stem vs BullMQ
sidebar_label: Stem vs BullMQ
sidebar_position: 1
slug: /comparisons/stem-vs-bullmq
---

This page is the canonical Stem comparison matrix for BullMQ-style features.
It focuses on capability parity, not API-level compatibility.

**As of:** February 24, 2026

## Status semantics

| Status | Meaning |
| --- | --- |
| `✓` | Functionally equivalent built-in capability exists in Stem. |
| `~` | Partial or non-isomorphic capability exists, but semantics differ from BullMQ/BullMQ-Pro. |
| `✗` | No built-in capability in Stem today. |

## Feature matrix

| BullMQ row | Stem | Rationale (with evidence) |
| --- | --- | --- |
| Backend | `✓` | Stem supports multiple backends/adapters (Redis, Postgres, SQLite, in-memory). See [Broker Overview](../brokers/overview.md) and [Developer Environment](../getting-started/developer-environment.md). |
| Observables | `✓` | Stem has built-in metrics, tracing, and lifecycle signals. See [Observability](../core-concepts/observability.md) and [Signals](../core-concepts/signals.md). |
| Group Rate Limit | `✓` | Stem supports group-scoped rate limiting via `TaskOptions.groupRateLimit`, `groupRateKey`, and `groupRateKeyHeader`. See [Rate Limiting](../core-concepts/rate-limiting.md). |
| Group Support | `✓` | Stem provides `Canvas.group` and `Canvas.chord` primitives. See [Canvas Patterns](../core-concepts/canvas.md). |
| Batches Support | `✓` | Stem exposes first-class batch APIs (`submitBatch`, `inspectBatch`) with durable batch lifecycle status. See [Canvas Patterns](../core-concepts/canvas.md). |
| Parent/Child Dependencies | `✓` | Stem supports dependency composition through chains, groups/chords, and workflow steps. See [Canvas Patterns](../core-concepts/canvas.md) and [Workflows](../core-concepts/workflows.md). |
| Deduplication (Debouncing) | `~` | `TaskOptions.unique` prevents duplicate enqueue claims, but semantics are lock/TTL-based rather than BullMQ-native dedupe APIs. See [Uniqueness](../core-concepts/uniqueness.md). |
| Deduplication (Throttling) | `~` | `uniqueFor` and lock TTL windows approximate throttling behavior, but are not a direct BullMQ equivalent. See [Uniqueness](../core-concepts/uniqueness.md). |
| Priorities | `✓` | Stem supports task priority and queue priority ranges. See [Tasks](../core-concepts/tasks.md) and [Routing](../core-concepts/routing.md). |
| Concurrency | `✓` | Workers support configurable concurrency and isolate pools. See [Workers](../workers/index.md) and [Worker Control](../workers/worker-control.md). |
| Delayed jobs | `✓` | Delayed execution is supported via enqueue options and broker scheduling. See [Quick Start](../getting-started/quick-start.md) and [Broker Overview](../brokers/overview.md). |
| Global events | `✓` | Stem exposes global lifecycle events through `StemSignals`, plus queue-scoped custom events through `QueueEvents`. See [Signals](../core-concepts/signals.md) and [Queue Events](../core-concepts/queue-events.md). |
| Rate Limiter | `✓` | Stem supports per-task rate limits with pluggable limiter backends. See [Rate Limiting](../core-concepts/rate-limiting.md). |
| Pause/Resume | `✓` | Stem provides queue pause/resume commands (`stem worker pause`, `stem worker resume`) and persistent pause state when a revoke store is configured. See [Worker Control](../workers/worker-control.md). |
| Sandboxed worker | `~` | Stem supports isolate-based execution boundaries, but this is not equivalent to BullMQ's Node child-process sandbox model. See [Worker Control](../workers/worker-control.md). |
| Repeatable jobs | `✓` | Stem Beat supports interval, cron, solar, and clocked schedules. See [Scheduler](../scheduler/index.md) and [Beat Scheduler Guide](../scheduler/beat-guide.md). |
| Atomic ops | `~` | Stem includes atomic behavior in specific stores/flows, but end-to-end transactional guarantees (for all enqueue/ack/result paths) are not universally built-in. See [Tasks idempotency guidance](../core-concepts/tasks.md#idempotency-checklist) and [Best Practices](../getting-started/best-practices.md). |
| Persistence | `✓` | Stem persists task/workflow/schedule state through pluggable backends/stores. See [Persistence & Stores](../core-concepts/persistence.md). |
| UI | `~` | Stem includes an experimental dashboard, not a fully mature operator UI parity target yet. See [Dashboard](../core-concepts/dashboard.md). |
| Optimized for | `~` | Stem is optimized for jobs/messages plus durable workflow orchestration, not only queue semantics. See [Core Concepts](../core-concepts/index.md) and [Workflows](../core-concepts/workflows.md). |

## Update policy

When this matrix changes:

1. Update the **As of** date.
2. Keep row names aligned with BullMQ terminology.
3. Update rationale links so every status remains auditable.

## BullMQ events parity notes

Stem supports the two common BullMQ event-listening styles:

| BullMQ concept | Stem equivalent |
| --- | --- |
| `QueueEvents` listeners | `QueueEvents` + `QueueEventsProducer` (queue-scoped custom events) |
| Custom queue events | `producer.emit(queue, eventName, payload, headers, meta)` |
| Worker-specific event listeners | `StemSignals` convenience APIs with `workerId` filters (`onWorkerReady`, `onWorkerInit`, `onTaskFailure`, `onControlCommandCompleted`, etc.) |
13 changes: 13 additions & 0 deletions .site/docs/core-concepts/canvas.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,19 @@ Groups fan out work and persist each branch in the result backend.

```

## Batches

Batches provide a first-class immutable submission API on top of durable group
state:

- `canvas.submitBatch(signatures)` returns a stable `batchId` and task ids.
- `canvas.inspectBatch(batchId)` returns aggregate lifecycle status
(`pending`, `running`, `succeeded`, `failed`, `cancelled`, `partial`).

```dart file=<rootDir>/../packages/stem/example/docs_snippets/lib/canvas_batch.dart#canvas-batch

```

## Chords

Chords combine a group with a callback. Once all body tasks succeed, the callback
Expand Down
21 changes: 18 additions & 3 deletions .site/docs/core-concepts/cli-control.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ manage workers, and operate schedules and routing.

## Remote control primer

The worker control commands (`ping`, `stats`, `inspect`, `revoke`, `shutdown`)
The worker control commands
(`ping`, `stats`, `inspect`, `revoke`, `shutdown`, `pause`, `resume`)
publish control messages into the broker. Each command uses a request id and
waits for replies on a per-request reply queue.

Expand All @@ -35,8 +36,9 @@ Inspect vs control semantics:

- **Inspect** (`ping`, `stats`, `inspect`) returns snapshots and does not
mutate worker state.
- **Control** (`revoke`, `shutdown`) persists intent and asks workers to change
behavior (terminate tasks or shut down).
- **Control** (`revoke`, `shutdown`, `pause`, `resume`) persists intent and asks
workers to change behavior (terminate tasks, shut down, or pause queue
consumption).

Payload highlights (as sent by the CLI):

Expand Down Expand Up @@ -87,6 +89,14 @@ stem worker stats --worker worker-a

```

```dart title="Pause queues" file=<rootDir>/../packages/stem/example/docs_snippets/lib/cli_control.dart#cli-control-worker-pause

```

```dart title="Resume queues" file=<rootDir>/../packages/stem/example/docs_snippets/lib/cli_control.dart#cli-control-worker-resume

```

```dart title="Apply schedules" file=<rootDir>/../packages/stem/example/docs_snippets/lib/cli_control.dart#cli-control-schedule-apply

```
Expand Down Expand Up @@ -154,6 +164,8 @@ stem worker ping
stem worker stats
stem worker revoke --task <id>
stem worker shutdown --mode warm
stem worker pause --queue default
stem worker resume --queue default
```

Expected output:
Expand Down Expand Up @@ -223,13 +235,16 @@ Use this table to sanity-check which connection strings are required:
| `stem worker status` | optional (follow) | optional (snapshot) | ❌ | ❌ | ❌ |
| `stem worker revoke` | ✅ | optional | ❌ | optional | ❌ |
| `stem worker shutdown` | ✅ | ❌ | ❌ | ❌ | ❌ |
| `stem worker pause/resume` | ✅ | ❌ | ❌ | optional | ❌ |
| `stem schedule apply/list/dry-run` | ❌ | ❌ | ✅ | ❌ | ❌ |
| `stem health` | ✅ | optional | ❌ | ❌ | ❌ |

Notes:

- The CLI resolves URLs from `STEM_BROKER_URL`, `STEM_RESULT_BACKEND_URL`,
`STEM_SCHEDULE_STORE_URL`, and `STEM_REVOKE_STORE_URL`.
- `STEM_REVOKE_STORE_URL` supports `redis://`, `postgres://`, `sqlite:///`,
`file:///`, and `memory://` targets.
- When a backend is “optional”, the command still runs but will skip that
slice of data (for example, worker heartbeats without a result backend).
- Schedule commands fall back to local schedule files when no schedule store
Expand Down
3 changes: 3 additions & 0 deletions .site/docs/core-concepts/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@ behavior before touching production.
- Worker lifecycle management, concurrency controls, and graceful shutdown.
- Beat scheduler for interval/cron/solar/clocked jobs.
- Canvas primitives (chains, groups, chords) for task composition.
- First-class batch submissions with durable aggregate status inspection.
- Lifecycle signals for instrumentation and integrations.
- Queue-scoped custom events via `QueueEventsProducer`/`QueueEvents`.
- Declarative routing across queues and broadcast channels.
- Result backends and progress reporting via `TaskContext`.

Expand Down Expand Up @@ -47,6 +49,7 @@ behavior before touching production.
- **[Namespaces](./namespaces.md)** – Isolate environments and tenants.
- **[Routing](./routing.md)** – Queue aliases, priorities, and broadcast channels.
- **[Signals](./signals.md)** – Lifecycle hooks for instrumentation and integrations.
- **[Queue Events](./queue-events.md)** – Publish/listen to queue-scoped custom events.
- **[Canvas Patterns](./canvas.md)** – Chains, groups, and chords for composing work.
- **[Observability](./observability.md)** – Metrics, traces, logging, and lifecycle signals.
- **[Persistence & Stores](./persistence.md)** – Result backends, schedule/lock stores, and revocation storage.
Expand Down
9 changes: 7 additions & 2 deletions .site/docs/core-concepts/persistence.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,13 +74,18 @@ Switch to Postgres with `PostgresScheduleStore.connect` / `PostgresLockStore.con

## Revoke store

Store revocations in Redis or Postgres so workers can honour `stem worker revoke`:
Store revocations in Redis/Postgres/SQLite so workers can honour
`stem worker revoke`:

```bash
export STEM_REVOKE_STORE_URL=postgres://postgres:postgres@localhost:5432/stem
```

```dart file=<rootDir>/../packages/stem/example/docs_snippets/lib/persistence.dart#persistence-revoke-store
```dart title="Postgres revoke store" file=<rootDir>/../packages/stem/example/docs_snippets/lib/persistence.dart#persistence-revoke-store

```

```dart title="SQLite revoke store" file=<rootDir>/../packages/stem/example/docs_snippets/lib/persistence.dart#persistence-revoke-store-sqlite

```

Expand Down
50 changes: 50 additions & 0 deletions .site/docs/core-concepts/queue-events.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
---
title: Queue Events
sidebar_label: Queue Events
sidebar_position: 9
slug: /core-concepts/queue-events
---

Stem supports queue-scoped custom events similar to BullMQ `QueueEvents` and
"custom events" patterns.

Use this when you need lightweight event streams for domain notifications
(`order.created`, `invoice.settled`) without creating task handlers.

## API Surface

- `QueueEventsProducer.emit(queue, eventName, payload, headers, meta)`
- `QueueEvents.start()` / `QueueEvents.close()`
- `QueueEvents.events` stream (all events for that queue)
- `QueueEvents.on(eventName)` stream (filtered by name)

All events are delivered as `QueueCustomEvent`, which implements `StemEvent`.

## Producer + Listener

```dart title="lib/queue_events.dart" file=<rootDir>/../packages/stem/example/docs_snippets/lib/queue_events.dart#queue-events-producer-listener

```

## Fan-out to Multiple Listeners

Multiple listeners on the same queue receive each emitted event.

```dart title="lib/queue_events.dart" file=<rootDir>/../packages/stem/example/docs_snippets/lib/queue_events.dart#queue-events-fanout

```

## Semantics

- Events are queue-scoped: listeners receive only events for their configured
queue.
- `on(eventName)` matches exact event names.
- `headers` and `meta` round-trip to listeners.
- Event names and queue names must be non-empty.
- Delivery follows the underlying broker's broadcast behavior for active
listeners (no historical replay API is built into `QueueEvents`).

## When to Use Queue Events vs Signals

- Use [Signals](./signals.md) for runtime lifecycle hooks (task/worker/scheduler/control).
- Use Queue Events for application-domain events you publish and consume.
21 changes: 21 additions & 0 deletions .site/docs/core-concepts/rate-limiting.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ Stem supports per-task rate limits via `TaskOptions.rateLimit` and a pluggable
`RateLimiter` interface. This lets you throttle hot handlers with a shared
Redis-backed limiter or custom driver.

Stem also supports group-scoped rate limits with `TaskOptions.groupRateLimit`
for shared quotas across multiple task types/tenants.

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

Expand Down Expand Up @@ -125,6 +128,8 @@ Run the `rate_limit_delay` example for a full demo:
- `100/m` — 100 tokens per minute
- `500/h` — 500 tokens per hour

`groupRateLimit` uses the same syntax.

## How it works

- The worker parses `rateLimit` for each task.
Expand All @@ -134,6 +139,22 @@ Run the `rate_limit_delay` example for a full demo:
worker’s retry strategy.
- If granted, the task executes immediately.

## Group rate limiting

Group rate limits share a limiter bucket across related tasks.

- `groupRateLimit`: limiter policy for the shared group bucket
- `groupRateKey`: optional static key (if omitted, Stem resolves from header)
- `groupRateKeyHeader`: header used when `groupRateKey` is not set
(default: `tenant`)
- `groupRateLimiterFailureMode` (default: `failOpen`):
- `failOpen`: continue execution if limiter backend fails
- `failClosed`: requeue/retry when limiter backend fails

```dart title="lib/rate_limiting.dart" file=<rootDir>/../packages/stem/example/docs_snippets/lib/rate_limiting.dart#rate-limit-group-task-options

```

## Redis-backed limiter example

The `example/rate_limit_delay` demo ships a Redis fixed-window limiter. It:
Expand Down
Loading
Loading