# Segmented history implementation

The durable implementation is in the macltok `segmented-history` feature
worktree. Its SDK dependency has been published at
`49a9cd1aff19df1de7e46710cb6c984a1bddc30e` on the SDK `segmented-history`
bookmark. The macltok manifest and lockfile are being updated to that revision.
The macltok feature bookmark will also be named `segmented-history`. Malice has
not yet been pinned to the macltok revision.

## Current behavior

Opening a room loads retained history and its gap rows, then fetches a bounded
recent window from a fresh server position. The recent fetch is capped at four
pages of 50 raw events. It preserves retained sections and their older cursors.
If it cannot establish continuity, it installs a durable gap between the
sections. It stops when continuity is established or accessible history ends.

`HistoryRepairCommand::FillGap` takes a room ID and stable gap transaction ID.
It fetches up to four backward pages of 50 events and advances only that gap's
saved cursor. Fill progress and events commit atomically. Concurrent history
operations for the room return `RoomBusy`. An empty page with an advancing
cursor keeps the gap. A repeated cursor pauses it. Accessible history ending
before the retained boundary marks the interval unavailable. Only reaching the
retained boundary joins sections. Already joined gaps are harmless to stale
callers.

The cache stores gap identity, boundary evidence, cursor, and status alongside
the encrypted linked event chunks. SQLite supports legacy token data and the
new metadata. IndexedDB stores the optional encrypted metadata field. Room
close and restart retain sections and fill cursors. Timeline snapshots and
incremental updates project gaps as `TimelineItem::HistoryGap`. Gap rows retain
their identity while status changes. Joined markers disappear and unavailable
intervals remain visible.

The recent fetch can complete while live sync continues. Cache installation
uses a candidate state and commits events, gap metadata, and cursor together.
Sync publication is serialized with gap installation. Existing loaded timelines
receive incremental insertions that preserve retained item identities and
pending sends. Loading history outside the current timeline window still needs
a reset.

Each SDK room cache owns a coordinator task. Recent callers share an in-flight
request. Cancelling one caller preserves the operation for other subscribers,
and cancelling all callers stops fetching before installation. Started commits
finish before the coordinator accepts conflicting work. No mutex guard crosses
the network request. The idle task holds only a weak cache reference, so it does
not keep a room cache alive. The SDK UI calls `wait_for_history` as a fence,
then takes an atomic event and gap snapshot while replacing its cache
subscription. The macltok watcher is stopped before its timeline reset is sent,
so old diffs cannot arrive after the reset. A mirror publication acknowledgement
completes before the command response is released. The room actor starts recent
loading in the background when the room opens, so opening does not wait for the
network request.

Limited sync results persist explicit gap markers in the event cache. Later sync
updates preserve those markers until the pagination chain reaches the retained
boundary. The integration uses the room's live timeline flush and watcher reset
path. It does not reseed the cache. Thread and pinned timelines remain open
through this update path.

## Public and wire shapes

`HistoryRepairCommand::FillGap { room_id, gap_id }` is the supported targeted
fill operation, sent through `ClientHandle::execute` as
`Command::HistoryRepair`. It returns `CommandOutput::Done` on success. The older
policy and staged repair command variants remain available for serde
compatibility, but room opening no longer consults the saved policy. `Continue`
is invalid because continuation must identify its gap. `GetPolicy` and
`SetPolicy` remain compatibility operations and do not control opening.

Targeted fill progress is published as a `HistoryRepairReport` with `gap_id`
set and `phase` set to `Bridging`. This report supplies the loading state while
the fill runs. Recent loading has its own room keyed `RecentHistoryState` with
`Idle`, `Loading`, `Loaded`, or `Failed` values. The mirror exposes that state in
`recent_history`, separately from gap continuity and repair progress.

`TimelineItem::HistoryGap` carries a `HistoryGap` with `id`,
`older_boundary`, `newer_boundary`, `lifecycle`, `continuity`, and `failure`.
Lifecycle values are `Idle`, `Fetching`, `Paused`, `Failed`, and `Unavailable`.
Continuity values are `Unresolved`, `Verified`, and `AccessibleStart`. The gap
ID is an owned transaction ID. Boundary values are optional owned event IDs.
Failure is an optional displayable string. `TimelineState.items` contains these
items in order with event rows. Mirror snapshots serialize the same item shape,
and incremental timeline changes carry it through the normal reset or changed
diff.

Serde enums use externally tagged variants. A command payload is shaped as
follows, with the outer `Command` adding its `HistoryRepair` tag.

```json
{
  "FillGap": {
    "room_id": "!room:example.org",
    "gap_id": "opaque-gap-transaction-id"
  }
}
```

The final history repair publication acknowledgement is internal, skipped by
serde, and not part of application wire data. The mirror applies the report
before acknowledging it, so command completion follows publication.

## Remaining publication and Malice work

Publish the SDK revision on the user's fork, then publish the macltok revision
that pins it. Update the Malice macltok dependency in both native and worker
manifests and regenerate the applicable lockfiles against that same revision.
The current worktree has not recorded a final publishable commit or revision
for these changes.

Malice should remove the saved policy and threshold controls from room opening.
Render each inline gap between its neighboring event sections. A fill should
preserve the visible event anchor, composer draft, reply or edit state, pending
sends, and encryption identity. Keep event-context windows independent from
live room history.

## Lost interval investigation

The reported failure showed retained events followed by newly sent messages,
with an omitted interval of roughly eight to ten hours presented as continuous.
Older pagination continued before the retained buffer, and a page refresh did
not recover the missing range. A refresh can reconnect to the existing
SharedWorker, so it does not prove that the engine restarted.

The earlier SDK cache path could discard a limited-sync boundary token when a
timeline was reported unlimited despite cached events. It could also discard a
token when all incoming events were duplicates. Both cases can hide a real
discontinuity from older pagination. Initial pagination could also finish
against retained history before fresh sync arrived. These are concrete failure
paths, but none has been confirmed as the production cause. The durable gaps
preserve discontinuities when observed. Startup, subscription, and sync
diagnostics are still needed to identify where this particular interval was
lost.

## Validation commands

Run native checks from `../macltok-history-repair` with the repository's Nix
toolchain.

```sh
nix develop --command cargo test --all-targets
nix develop --command cargo test --all-targets --features sqlite
nix develop --command cargo clippy --all-targets -- -D warnings
nix develop --command cargo clippy --all-targets --features sqlite -- -D warnings
```

For the browser worker runtime, use the web devshell. The harness starts a local
homeserver and runs both worker targets in Firefox.

```sh
nix develop .#web -c nu harness/web/run.nu
```

The flake checks include native and WebAssembly compilation with IndexedDB and
the web worker runtime. `nix flake check` is the aggregate gate and remains
pending.

Seven focused SDK recent-history tests and seven segmented-history tests pass.
The SDK WebAssembly compile passes with IndexedDB enabled. The SDK UI timeline
checks pass with 123 unit tests, seven room unit tests, and 14 timed integration
tests. Two headless Firefox tests pass for the IndexedDB shared gap storage
contract with plaintext and encrypted stores. The full flake gate remains
pending. Complete worker restart and encrypted room recovery have not been
tested.