incidentd - Architecture

incidentd was built around a simple idea:
strong atomic guarantees + durable eventual consistency, independent of where inputs come from or where side-effects go.

It must be incident-free.

incidentd is a standalone service that manages an incident end-to-end: from declaration to resolution. It owns state, notifications, AI suggestions, status page updates, and post-incident analysis.

From day one, it was designed to feed context into AI agents. That means data lives close, and reads/writes are fast.

Platform

incidentd runs on Cloudflare Workers, using 3 primitives:

• Durable Objects -> per-incident transactional state
• D1 -> queryable indexing and listing
• Workflows -> async side effects and AI jobs

Working on Cloudflare has been surprisingly good: great DX, trivial deployment and low cost.

The only real downside is ISPs blocking thanks to a spectacularly bad ruling that gave power to LaLiga to control the internet. (irrelevant in this case since this is server-to-server)

The Durable Object

Each incident maps to a single Durable Object.

This is the source of truth for:

• Event timeline (event_log)
• Derived state: status, severity, assignee, metadata

Every mutation follows the same pattern:

1. Start transaction
2. Update state
3. Append event (published_at = NULL)
4. Schedule alarm
5. Commit

This is the outbox pattern.

The core idea behind incidentd is:

A single-threaded, transactional system that gives you atomicity and reliable async delivery, without distributed coordination.

Alarm-based delivery

The DO alarm drains unpublished events:

for each event where published_at IS NULL and attempts < 3:
    dispatch to IncidentWorkflow
    set published_at = now

If dispatch fails, it retries. After 3 attempts, the event is marked failed but acknowledged.
This gives at-least-once delivery with bounded retries.

The same alarm also drives agent scheduling. A single method decideAlarmAction() decides what to do next:

• dispatch events
• trigger an agent turn
• schedule cleanup
• or do nothing

This keeps the reliability story local: one place decides what must happen next, and one durable log says what already happened.

The dispatcher

IncidentWorkflow bridges the incident Durable Object and the senders. It’s a long-lived workflow that processes events in order:

dispatch first event
while incident is not resolved or declined:
    wait for next event
    dispatch it

Each event is fanned out to all senders using Promise.allSettled.

• A failing sender doesn’t block others
• Failures are logged, not fatal

Because delivery is at-least-once, senders should be idempotent (incident_id + event_id).

Events are committed sequentially in the DO, and dispatched in the same order.
No races, no inconsistent external state.

Why not Queues?

Queues have a classic problem: dual-writes. Update state in one system, enqueue in another. If enqueue fails, the event is lost.

It can be fixed (outbox tables, polling, distributed transactions), but complexity explodes.

Here, the outbox lives inside the Durable Object, and dispatch goes directly to a workflow.

Also:

Workflows are basically queues with better ergonomics

So I use them.

IDs and identifiers

• Internal IDs: newUniqueId() → stored in D1
• DO resolution: idFromString
• External IDs: Slack, dashboard, etc. → stored as routing metadata

Originally I used idFromName (e.g. Slack thread ID), but that made IDs non-reusable after resolution.

The plugin model

The system is structured as: receivers → handler → durable object → dispatcher → senders

Each stage is decoupled

1. Receivers -> validate + normalize input
2. The handler -> pushes events to the DO
3. DO -> commits state + events atomically and schedules dispatch
4. The dispatcher -> delivers events
5. Senders -> produce side-effects

Adapters isolate integrations:

src/adapters/<name>/receiver/
src/adapters/<name>/sender/

Adding a new integration means adding an adapter, not touching core logic.

Current adapters

Dashboard (src/adapters/dashboard/)

• Receiver: REST API
• Sender: writes to D1

Slack (src/adapters/slack/)

• Receiver: events, interactions
• Sender: messages, threads, channels

Status page (src/dispatcher/status-page.ts)

• Sender-only: persists affections to external DB

Each adapter handles idempotency in its own way.

The AI agent

Each incident runs an agent that watches the timeline and generates suggestions.

It helps:

• resolve the incident
• keep stakeholders informed

The agent runs asynchronously.
It reads from the timeline and writes back into it.

It is more inspired by Codex than by a classic tool-calling loop: each turn is a IncidentAgentTurnWorkflow run.

Debouncing

Not every event should trigger a turn.

• First turn: delayed 60s (let the incident stabilize)
• Subsequent turns: 13s debounce window

Execution

Each turn:

1. Fetch snapshot (state + events)
2. Call LLM with tools
3. Optionally trigger context agents
4. Persist suggestions as events
5. Post to Slack

Suggestions are stored as internal events, so future turns have memory and avoid repetition.

Context agents

Context agents are separate Durable Objects that run background investigations.

That keeps the incident DO focused on coordination, not research.

They are decoupled to:

• run asynchronously
• avoid polluting the main incident context

Base agent

Context agents extend AgentBase, which provides:

• SQLite storage -> (steps, contexts)
• Summarization -> keeps token usage bounded
• Alarm-driven execution
• Lazy initialization

Bidirectional RPC

The incident DO and context agents communicate in both directions:

-> provider:

• addContext: pushes new timeline events for background processing
• addPrompt: handles user questions from Slack, pulling the provider’s latest findings

Provider ->:

• getAgentContext: pulls the full incident snapshot to inform investigation
• recordAgentInsightEvent: writes insight events (SIMILAR_INCIDENT, GITHUB_COMMIT) without published_at, so they flow through the normal alarm/dispatch cycle and trigger new agent turns

Current providers

• SimilarIncidentsAgent -> searches past incidents for similar patterns.
• GitHubCommitsAgent -> surfaces relevant commits

Future ones:

• AWS investigator (via CLI, on Container-enabled Durable Objects)
• DataDog (via MCP)

Prompt workflow

Users can directly ask the agent questions. Handled by IncidentPromptWorkflow:

1. LLM decides what to do
2. If needed, calls context agents
3. Executes action or responds

A :fire: reaction shows progress.

Expected usage is command-like (“resolve incident”, “post status page update”), making it easy for untrained responders.

Reliability

Everything optimizes for one goal:

never lose an event

• Acknowledgement happens after persistence
• State + event are atomic
• Delivery is retried
• Senders are isolated

So far: ** 0 lost events**, with low double-digit millisecond latency.

Folder structure

- Adapters: src/adapters/*
- Handler: src/handler
- Core DO: src/core/incident.ts
- Alarm: src/core/incident/alarm.ts
- Dispatcher: src/dispatcher/workflow.ts
- Agents: src/agent/providers/

---

The only part I’m still unsure about is how context agents will interact as their number grows.

We’ll find out.