atagen

macltok

public
Star0
Watch0
d9b96952ec6003585d618d852b489c48fc386d5d
3
ssh://brewery@pls.lobotomise.me:33/atagen/macltok.git

Blame: docs/segmented-history.md

d9b96952ec60atagen2026-10-25 09:07 UTC1
# segmented history
d9b96952ec60atagen2026-10-25 09:07 UTC2
d9b96952ec60atagen2026-10-25 09:07 UTC3
a room's history is sections of events with durable gaps between them. the sdk
d9b96952ec60atagen2026-10-25 09:07 UTC4
event cache keeps the sections, gap identity, boundaries and each gap's cursor,
d9b96952ec60atagen2026-10-25 09:07 UTC5
encrypted at rest, across room close and restart. macltok shows gaps inline and
d9b96952ec60atagen2026-10-25 09:07 UTC6
fills them on request.
d9b96952ec60atagen2026-10-25 09:07 UTC7
d9b96952ec60atagen2026-10-25 09:07 UTC8
## opening a room
d9b96952ec60atagen2026-10-25 09:07 UTC9
d9b96952ec60atagen2026-10-25 09:07 UTC10
opening loads retained history and its gaps, then starts a recent load in the
d9b96952ec60atagen2026-10-25 09:07 UTC11
background, so opening never waits on the network. the recent load fetches up
d9b96952ec60atagen2026-10-25 09:07 UTC12
to four pages of 50 events from a fresh server position. if it reaches the
d9b96952ec60atagen2026-10-25 09:07 UTC13
newest retained event the sections join. if it doesn't, a gap is left between
d9b96952ec60atagen2026-10-25 09:07 UTC14
the fresh window and what was retained.
d9b96952ec60atagen2026-10-25 09:07 UTC15
d9b96952ec60atagen2026-10-25 09:07 UTC16
`Event::RecentHistoryChanged` reports it, one event per edge.
d9b96952ec60atagen2026-10-25 09:07 UTC17
d9b96952ec60atagen2026-10-25 09:07 UTC18
- `Loading` as it starts
d9b96952ec60atagen2026-10-25 09:07 UTC19
- `Loaded`, `Cancelled` or `Failed(message)` as it settles
d9b96952ec60atagen2026-10-25 09:07 UTC20
- `Idle` when the room closes, whatever was happening
d9b96952ec60atagen2026-10-25 09:07 UTC21
d9b96952ec60atagen2026-10-25 09:07 UTC22
`Failed` stays until `HistoryRepairCommand::RetryRecent` or a reopen. a failed
d9b96952ec60atagen2026-10-25 09:07 UTC23
load falls back to paginating `EngineConfig::initial_history_limit` events, so
d9b96952ec60atagen2026-10-25 09:07 UTC24
the room isn't left empty.
d9b96952ec60atagen2026-10-25 09:07 UTC25
d9b96952ec60atagen2026-10-25 09:07 UTC26
## filling a gap
d9b96952ec60atagen2026-10-25 09:07 UTC27
d9b96952ec60atagen2026-10-25 09:07 UTC28
`HistoryRepairCommand::FillGap { room_id, gap_id }` walks one gap backwards, up
d9b96952ec60atagen2026-10-25 09:07 UTC29
to four pages of 50, each page committing its events and the gap's new cursor
d9b96952ec60atagen2026-10-25 09:07 UTC30
together. `Event::HistoryRepairChanged` carries a `HistoryRepairReport`, once
d9b96952ec60atagen2026-10-25 09:07 UTC31
with phase `Bridging` as the fill starts and once more as it settles.
d9b96952ec60atagen2026-10-25 09:07 UTC32
d9b96952ec60atagen2026-10-25 09:07 UTC33
- `Ready`, continuity `Verified`. the older section was reached and the gap is gone
d9b96952ec60atagen2026-10-25 09:07 UTC34
- `Paused` with a stop. `Budget` after four pages, `CursorStalled` when the
d9b96952ec60atagen2026-10-25 09:07 UTC35
  server repeats a cursor, `HistoryUnavailable` (continuity `AccessibleStart`)
d9b96952ec60atagen2026-10-25 09:07 UTC36
  when accessible history ends first, `Cancelled`
d9b96952ec60atagen2026-10-25 09:07 UTC37
- `Paused` with a `failure`, and the gap row goes `Failed`
d9b96952ec60atagen2026-10-25 09:07 UTC38
d9b96952ec60atagen2026-10-25 09:07 UTC39
a gap that is already joined or unavailable answers at once and counts no page.
d9b96952ec60atagen2026-10-25 09:07 UTC40
d9b96952ec60atagen2026-10-25 09:07 UTC41
## one operation per room
d9b96952ec60atagen2026-10-25 09:07 UTC42
d9b96952ec60atagen2026-10-25 09:07 UTC43
a room runs one history operation at a time. `RetryRecent` joins a recent load
d9b96952ec60atagen2026-10-25 09:07 UTC44
that is already running. everything else that overlaps, a fill during a recent
d9b96952ec60atagen2026-10-25 09:07 UTC45
load, a recent load during a fill, a second fill, answers `RoomBusy`.
d9b96952ec60atagen2026-10-25 09:07 UTC46
d9b96952ec60atagen2026-10-25 09:07 UTC47
`HistoryRepairCommand::Cancel` stops whatever is running. its own answer is
d9b96952ec60atagen2026-10-25 09:07 UTC48
`Ok`, the cancelled command answers `Interrupted`, and the published outcome is
d9b96952ec60atagen2026-10-25 09:07 UTC49
`Cancelled`, never a failure. pages already committed stay.
d9b96952ec60atagen2026-10-25 09:07 UTC50
d9b96952ec60atagen2026-10-25 09:07 UTC51
## the timeline
d9b96952ec60atagen2026-10-25 09:07 UTC52
d9b96952ec60atagen2026-10-25 09:07 UTC53
gaps appear in `TimelineState.items` as `TimelineItem::HistoryGap`, between the
d9b96952ec60atagen2026-10-25 09:07 UTC54
sections they separate. a row keeps its id while its lifecycle changes, and is
d9b96952ec60atagen2026-10-25 09:07 UTC55
removed once the gap joins. joined gaps never appear.
d9b96952ec60atagen2026-10-25 09:07 UTC56
d9b96952ec60atagen2026-10-25 09:07 UTC57
history lands as `TimelineChanged` diffs, not a `TimelineReset`. the sdk
d9b96952ec60atagen2026-10-25 09:07 UTC58
currently replays the whole live timeline when it applies committed history,
d9b96952ec60atagen2026-10-25 09:07 UTC59
so those diffs can span the timeline until it gets a non-replacing fence.
d9b96952ec60atagen2026-10-25 09:07 UTC60
d9b96952ec60atagen2026-10-25 09:07 UTC61
a command's answer comes after its outcome is published, and after the mirror
d9b96952ec60atagen2026-10-25 09:07 UTC62
has applied it when one is attached.
d9b96952ec60atagen2026-10-25 09:07 UTC63
d9b96952ec60atagen2026-10-25 09:07 UTC64
## known holes
d9b96952ec60atagen2026-10-25 09:07 UTC65
d9b96952ec60atagen2026-10-25 09:07 UTC66
a limited sync for a room with no events in memory records a plain gap instead
d9b96952ec60atagen2026-10-25 09:07 UTC67
of a history gap (review 2026-10-25 S1), so an interval can still render as
d9b96952ec60atagen2026-10-25 09:07 UTC68
continuous. both `prev_batch` drops in the sdk's `handle_sync` remain.
d9b96952ec60atagen2026-10-25 09:07 UTC69
d9b96952ec60atagen2026-10-25 09:07 UTC70
## wire shapes
d9b96952ec60atagen2026-10-25 09:07 UTC71
d9b96952ec60atagen2026-10-25 09:07 UTC72
serde enums are externally tagged.
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC73
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC74
```json
d9b96952ec60atagen2026-10-25 09:07 UTC75
{ "HistoryRepair": { "FillGap": { "room_id": "!room:example.org", "gap_id": "opaque" } } }
d9b96952ec60atagen2026-10-25 09:07 UTC76
{ "HistoryRepair": { "RetryRecent": { "room_id": "!room:example.org" } } }
d9b96952ec60atagen2026-10-25 09:07 UTC77
{ "HistoryRepair": { "Cancel": { "room_id": "!room:example.org" } } }
9eabb6b7587datagen2026-10-25 07:08 UTC78
```
9eabb6b7587datagen2026-10-25 07:08 UTC79
d9b96952ec60atagen2026-10-25 09:07 UTC80
a gap row.
9eabb6b7587datagen2026-10-25 07:08 UTC81
9eabb6b7587datagen2026-10-25 07:08 UTC82
```json
9eabb6b7587datagen2026-10-25 07:08 UTC83
{
9eabb6b7587datagen2026-10-25 07:08 UTC84
  "HistoryGap": {
d9b96952ec60atagen2026-10-25 09:07 UTC85
    "id": "opaque",
9eabb6b7587datagen2026-10-25 07:08 UTC86
    "older_boundary": "$older:example.org",
9eabb6b7587datagen2026-10-25 07:08 UTC87
    "newer_boundary": "$newer:example.org",
9eabb6b7587datagen2026-10-25 07:08 UTC88
    "lifecycle": "Idle",
9eabb6b7587datagen2026-10-25 07:08 UTC89
    "failure": null
9eabb6b7587datagen2026-10-25 07:08 UTC90
  }
9eabb6b7587datagen2026-10-25 07:08 UTC91
}
9eabb6b7587datagen2026-10-25 07:08 UTC92
```
9eabb6b7587datagen2026-10-25 07:08 UTC93
d9b96952ec60atagen2026-10-25 09:07 UTC94
lifecycle is `Idle`, `Paused`, `Failed` or `Unavailable`. `failure` is set with
d9b96952ec60atagen2026-10-25 09:07 UTC95
`Failed`.
d9b96952ec60atagen2026-10-25 09:07 UTC96
d9b96952ec60atagen2026-10-25 09:07 UTC97
a settled fill.
9eabb6b7587datagen2026-10-25 07:08 UTC98
9eabb6b7587datagen2026-10-25 07:08 UTC99
```json
9eabb6b7587datagen2026-10-25 07:08 UTC100
{
d9b96952ec60atagen2026-10-25 09:07 UTC101
  "room_id": "!room:example.org",
d9b96952ec60atagen2026-10-25 09:07 UTC102
  "gap_id": "opaque",
d9b96952ec60atagen2026-10-25 09:07 UTC103
  "pages": 2,
d9b96952ec60atagen2026-10-25 09:07 UTC104
  "events": 63,
d9b96952ec60atagen2026-10-25 09:07 UTC105
  "phase": "Ready",
d9b96952ec60atagen2026-10-25 09:07 UTC106
  "continuity": "Verified",
d9b96952ec60atagen2026-10-25 09:07 UTC107
  "stop": null,
d9b96952ec60atagen2026-10-25 09:07 UTC108
  "failure": null
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC109
}
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC110
```
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC111
d9b96952ec60atagen2026-10-25 09:07 UTC112
the mirror keeps `State.recent_history`, a room without an entry being idle,
d9b96952ec60atagen2026-10-25 09:07 UTC113
and `State.repairs`, the last fill report per room. closing a room drops both.