macltok

# Segmented history implementation

The durable implementation is in the macltok `segmented-history` feature

worktree. The combined SDK revision

`9b2c17fb10703d90b177b02588912a34cecfa11f` is published on its

`segmented-history` bookmark. It includes the reaction compatibility fix from

`c0a` and segmented history from `49a9`. The macltok history branch includes

pack fixes from `e4bb841a` and the `macltok-pack-fixes` worktree through

`d1313f47`. All nine SDK pins and both lockfiles use the combined SDK revision.

The unified macltok bookmark is named `segmented-history`.

## Current behavior

Opening a room loads retained history and its gap rows, then fetches a bounded

recent window from a fresh server position. The recent fetch is capped at four

pages of 50 raw events. It preserves retained sections and their older cursors.

If it cannot establish continuity, it installs a durable gap between the

sections. It stops when continuity is established or accessible history ends.

`HistoryRepairCommand::FillGap` takes a room ID and stable gap transaction ID.

It fetches up to four backward pages of 50 events and advances only that gap's

saved cursor. Fill progress and events commit atomically. Concurrent history

operations for the room return `RoomBusy`. An empty page with an advancing

cursor keeps the gap. A repeated cursor pauses it. Accessible history ending

before the retained boundary marks the interval unavailable. Only reaching the

retained boundary joins sections. Already joined gaps are harmless to stale

callers.

The cache stores gap identity, boundary evidence, cursor, and status alongside

the encrypted linked event chunks. SQLite supports legacy token data and the

new metadata. IndexedDB stores the optional encrypted metadata field. Room

close and restart retain sections and fill cursors. Timeline snapshots and

incremental updates project gaps as `TimelineItem::HistoryGap`. Gap rows retain

their identity while status changes. Joined markers disappear and unavailable

intervals remain visible.

The recent fetch can complete while live sync continues. Cache installation

uses a candidate state and commits events, gap metadata, and cursor together.

Sync publication is serialized with gap installation. Existing loaded timelines

receive incremental insertions that preserve retained item identities and

pending sends. Loading history outside the current timeline window still needs

a reset.

Each SDK room cache owns a coordinator task. Recent callers share an in-flight

request. Cancelling one caller preserves the operation for other subscribers,

and cancelling all callers stops fetching before installation. Started commits

finish before the coordinator accepts conflicting work. No mutex guard crosses

the network request. The idle task holds only a weak cache reference, so it does

not keep a room cache alive. The SDK UI calls `wait_for_history` as a fence,

then takes an atomic event and gap snapshot while replacing its cache

subscription. The macltok watcher is stopped before its timeline reset is sent,

so old diffs cannot arrive after the reset. A mirror publication acknowledgement

completes before the command response is released. The room actor starts recent

loading in the background when the room opens, so opening does not wait for the

network request.

Limited sync results persist explicit gap markers in the event cache. Later sync

updates preserve those markers until the pagination chain reaches the retained

boundary. The integration uses the room's live timeline flush and watcher reset

path. It does not reseed the cache. Thread and pinned timelines remain open

through this update path.

## Public and wire shapes

`ClientHandle::fill_history_gap(room_id, gap_id)` is the convenience method for

the targeted fill. It returns `Result<(), Error>`. The underlying command is

`HistoryRepairCommand::FillGap { room_id, gap_id }`, sent as

`Command::HistoryRepair`. The older policy and staged repair command variants

remain available for serde compatibility, but room opening no longer consults

the saved policy. `Continue` is invalid because continuation must identify its

gap. `GetPolicy` and `SetPolicy` remain compatibility operations and do not

control opening.

The inline gap lifecycle value `Fetching` is reserved and is not currently

emitted by timeline projection. During a targeted fill, loading is signaled by

a `HistoryRepairReport` with `gap_id` set and `phase` set to `Bridging`. Recent

loading has its own room keyed `RecentHistoryState` with

`Idle`, `Loading`, `Loaded`, or `Failed` values. The mirror exposes that state in

`recent_history`, separately from gap continuity and repair progress.

`TimelineItem::HistoryGap` carries a `HistoryGap` with `id`,

`older_boundary`, `newer_boundary`, `lifecycle`, `continuity`, and `failure`.

Lifecycle values are `Idle`, `Fetching`, `Paused`, `Failed`, and `Unavailable`.

Continuity values are `Unresolved`, `Verified`, and `AccessibleStart`. The gap

ID is an owned transaction ID. Boundary values are optional owned event IDs.

Failure is an optional displayable string. `TimelineState.items` contains these

items in order with event rows. Mirror snapshots serialize the same item shape,

and incremental timeline changes carry it through the normal reset or changed

diff.

Serde enums use externally tagged variants. A command payload is shaped as

follows, with the outer `Command` adding its `HistoryRepair` tag.

```json

{

  "HistoryRepair": {

    "FillGap": {

      "room_id": "!room:example.org",

      "gap_id": "opaque-gap-transaction-id"

    }

  }

}

```

A gap row in `TimelineState.items` has the following shape.

```json

{

  "HistoryGap": {

    "id": "opaque-gap-transaction-id",

    "older_boundary": "$older:example.org",

    "newer_boundary": "$newer:example.org",

    "lifecycle": "Idle",

    "continuity": "Unresolved",

    "failure": null

  }

}

```

The mirror snapshot stores recent loading separately by room ID. This fragment

shows the serialized `State.recent_history` field.

```json

{

  "recent_history": {

    "!room:example.org": "Loading"

  }

}

```

The final history repair publication acknowledgement is internal, skipped by

serde, and not part of application wire data. The mirror applies the report

before acknowledging it, so command completion follows publication.

## Malice integration

Pin the published macltok `segmented-history` revision in Malice's native and

worker manifests and regenerate their lockfiles. The SDK dependency is already

pinned by macltok.

Malice should remove the saved policy and threshold controls from room opening.

Render each inline gap between its neighboring event sections. A fill should

preserve the visible event anchor, composer draft, reply or edit state, pending

sends, and encryption identity. Keep event-context windows independent from

live room history.

## Lost interval investigation

The reported failure showed retained events followed by newly sent messages,

with an omitted interval of roughly eight to ten hours presented as continuous.

Older pagination continued before the retained buffer, and a page refresh did

not recover the missing range. A refresh can reconnect to the existing

SharedWorker, so it does not prove that the engine restarted.

The earlier SDK cache path could discard a limited-sync boundary token when a

timeline was reported unlimited despite cached events. It could also discard a

token when all incoming events were duplicates. Both cases can hide a real

discontinuity from older pagination. Initial pagination could also finish

against retained history before fresh sync arrived. These are concrete failure

paths, but none has been confirmed as the production cause. The durable gaps

preserve discontinuities when observed. Startup, subscription, and sync

diagnostics are still needed to identify where this particular interval was

lost.

## Validation commands

Run native checks from `../macltok-history-repair` with the repository's Nix

toolchain.

```sh

nix develop --command cargo test --all-targets

nix develop --command cargo test --all-targets --features sqlite

nix develop --command cargo clippy --all-targets -- -D warnings

nix develop --command cargo clippy --all-targets --features sqlite -- -D warnings

```

For the browser worker runtime, use the web devshell. The harness starts a local

homeserver and runs both worker targets in Firefox.

```sh

nix develop .#web -c nu harness/web/run.nu

```

The flake checks include native and WebAssembly compilation with IndexedDB and

the web worker runtime. The default native gate passed all 86 tests. The

aggregate `nix flake check` run stopped during the SQLite and mirror build.

The remaining aggregate gates were waived for this handoff and have not passed.

The focused native history repair and engine suites passed all 11 tests with

SQLite, mirror, and serde enabled against the published combined SDK revision.

Seven focused SDK recent-history tests and seven segmented-history tests pass.

The SDK WebAssembly compile passes with IndexedDB enabled. SDK UI timeline

checks pass with 123 unit tests. Separate SDK room checks pass with seven unit

tests and 14 timed integration tests. Two headless Firefox tests pass for the

IndexedDB shared gap storage contract with plaintext and encrypted stores. The

full SQLite and mirror gate, aggregate WebAssembly gate, and browser worker

runtime gate remain unverified. Complete worker restart and encrypted room

recovery remain integration acceptance checks.