Lesson 14 — Raw before parse

Outcome

Persist every fake/provider raw frame before normalization and make raw history robust to a partial final write.

Why this comes now

Once real protocols arrive, parser bugs and version drift are inevitable. Keeping raw evidence makes them diagnosable and re-normalizable.

Understand

The safe order is: receive frame → append durable raw record → obtain RawRef → parse/normalize. Raw storage is append-only and lives outside SQLite. SQLite stores queryable normalized state.

Frame the raw JSONL with Forge metadata such as format version, sequence, timestamp, source, and provider payload. JSONL is simple, but it still needs a version contract.

Build the real project

1. Add RawLog interface and local file implementation under XDG data path.
2. Define a framed line schema with formatVersion/sequence/source/receivedAt/payload.
3. Append with ordering per Run and return a RawRef.
4. Change fake adapter ingestion so normalization cannot occur without a RawRef.
5. Add reader/recovery behavior that ignores or reports a corrupt/truncated final line without losing earlier valid lines.
6. Test parser failure after raw append.

Completion gate

Every normalized event test can point to non-empty raw evidence. A simulated truncated final line does not make valid earlier history unreadable.

Pitfalls to avoid

Do not keep the only raw copy in memory. Do not put giant transcripts into SQLite. Do not pretend “JSONL” means schema evolution is solved.

References

Forge ADR direction in DATA_MODEL.md; newline-delimited JSON overview: https://jsonlines.org/

Checkpoint

Record the on-disk raw format version in docs if implementation differs from the proposed example.