Durable Operations and Recovery

External systems cannot participate in the same transaction as SQLite. Forge therefore records durable intent before performing important side effects.

The problem

This is unsafe if treated as one indivisible action:

set Task = WORKING
start provider
create worktree
save run row

The application can crash between any two lines. The filesystem/provider may have changed while SQLite did not, or SQLite may claim progress that never happened.

The pattern

Application command
      │
      ▼
SQLite transaction
┌──────────────────────────────┐
│ validate domain transition   │
│ update domain state          │
│ insert durable Operation     │
└─────────────┬────────────────┘
              │ commit
              ▼
Operation worker
      │
      ▼
external side effect
      │
      ▼
record outcome / emit follow-up command

The command being accepted means intent was committed, not that the external action finished.

Operation properties

Every handler should consider:

• idempotency: what happens if the same intent is attempted twice?
• reconciliation: if Forge restarts after the external system changed, how can we discover reality?
• cancellation: what does cancellation mean before, during, and after the action?
• retry safety: which errors are transient and which require human intervention?
• ownership: which subsystem is allowed to perform the side effect?

Examples

Create worktree

1. Transaction creates CREATE_WORKTREE Operation.
2. Worker asks Git/Worktree service whether the target worktree already exists.
3. If it already exists in the expected state, treat as reconciled success.
4. Otherwise create it, verify it, record outcome.

Start agent

Starting an agent is harder because a provider may start successfully and Forge may crash before persisting the external session identity. The adapter strategy must document whether startup can be reconciled. The Run stays in a state that does not falsely promise activity until the outcome is known.

Run verification

Create the Verification Run and Operation against an exact revision. The worker executes through the controlled process boundary. Completion updates that run only; it does not automatically approve the Task.

Recovery on core startup

Startup should eventually perform a reconciliation pass before normal scheduling:

open DB
  ↓
run migrations
  ↓
load unfinished Operations / Runs / Worktrees
  ↓
reconcile external reality
  ↓
mark recoverable operations pending or settled
  ↓
start normal workers/RPC

Do not build the full recovery matrix in Milestone 0. The course introduces it gradually, but the architecture keeps a place for it from the beginning.

Worker design

Use bounded workers. Avoid unbounded goroutine creation. Tests need a real notion of quiescence: an empty queue does not prove the current item finished. Provide a Drain/test hook that waits for both queued and in-flight work.

Retry model

Do not implement a generic “retry everything three times” policy.

Classify by operation. For example:

• temporary filesystem lock may be retryable;
• invalid repository path is not;
• provider startup timeout might be retryable with limits;
• merge conflict requires user/action workflow rather than blind retry.

Persist attempt count and enough structured diagnostics to explain the failure.

What Operations are not

This is not a distributed job queue and not full event sourcing. SQLite remains the source of current application state. Operations are the durable bridge between transactional intent and non-transactional effects.