atagen

macltok

public
Star0
Watch0
5b585cd970fd8c14c399754bfd12500c872c8c08
3
ssh://brewery@pls.lobotomise.me:33/atagen/macltok.git

Blame: docs/segmented-history.md

d9b96952ec60atagen2026-10-25 09:07 UTC1
# segmented history
d9b96952ec60atagen2026-10-25 09:07 UTC2
d9b96952ec60atagen2026-10-25 09:07 UTC3
a room's history is sections of events with durable gaps between them. the sdk
d9b96952ec60atagen2026-10-25 09:07 UTC4
event cache keeps the sections, gap identity, boundaries and each gap's cursor,
d9b96952ec60atagen2026-10-25 09:07 UTC5
encrypted at rest, across room close and restart. macltok shows gaps inline and
d9b96952ec60atagen2026-10-25 09:07 UTC6
fills them on request.
d9b96952ec60atagen2026-10-25 09:07 UTC7
d9b96952ec60atagen2026-10-25 09:07 UTC8
## opening a room
d9b96952ec60atagen2026-10-25 09:07 UTC9
d9b96952ec60atagen2026-10-25 09:07 UTC10
opening loads retained history and its gaps, then starts a recent load in the
d9b96952ec60atagen2026-10-25 09:07 UTC11
background, so opening never waits on the network. the recent load fetches up
d9b96952ec60atagen2026-10-25 09:07 UTC12
to four pages of 50 events from a fresh server position. if it reaches the
d9b96952ec60atagen2026-10-25 09:07 UTC13
newest retained event the sections join. if it doesn't, a gap is left between
d9b96952ec60atagen2026-10-25 09:07 UTC14
the fresh window and what was retained.
d9b96952ec60atagen2026-10-25 09:07 UTC15
d9b96952ec60atagen2026-10-25 09:07 UTC16
`Event::RecentHistoryChanged` reports it, one event per edge.
d9b96952ec60atagen2026-10-25 09:07 UTC17
d9b96952ec60atagen2026-10-25 09:07 UTC18
- `Loading` as it starts
d9b96952ec60atagen2026-10-25 09:07 UTC19
- `Loaded`, `Cancelled` or `Failed(message)` as it settles
d9b96952ec60atagen2026-10-25 09:07 UTC20
- `Idle` when the room closes, whatever was happening
d9b96952ec60atagen2026-10-25 09:07 UTC21
d9b96952ec60atagen2026-10-25 09:07 UTC22
`Failed` stays until `HistoryRepairCommand::RetryRecent` or a reopen. a failed
d9b96952ec60atagen2026-10-25 09:07 UTC23
load falls back to paginating `EngineConfig::initial_history_limit` events, so
d9b96952ec60atagen2026-10-25 09:07 UTC24
the room isn't left empty.
d9b96952ec60atagen2026-10-25 09:07 UTC25
d9b96952ec60atagen2026-10-25 09:07 UTC26
## filling a gap
d9b96952ec60atagen2026-10-25 09:07 UTC27
d9b96952ec60atagen2026-10-25 09:07 UTC28
`HistoryRepairCommand::FillGap { room_id, gap_id }` walks one gap backwards, up
d9b96952ec60atagen2026-10-25 09:07 UTC29
to four pages of 50, each page committing its events and the gap's new cursor
d9b96952ec60atagen2026-10-25 09:07 UTC30
together. `Event::HistoryRepairChanged` carries a `HistoryRepairReport`, once
d9b96952ec60atagen2026-10-25 09:07 UTC31
with phase `Bridging` as the fill starts and once more as it settles.
d9b96952ec60atagen2026-10-25 09:07 UTC32
d9b96952ec60atagen2026-10-25 09:07 UTC33
- `Ready`, continuity `Verified`. the older section was reached and the gap is gone
d9b96952ec60atagen2026-10-25 09:07 UTC34
- `Paused` with a stop. `Budget` after four pages, `CursorStalled` when the
d9b96952ec60atagen2026-10-25 09:07 UTC35
  server repeats a cursor, `HistoryUnavailable` (continuity `AccessibleStart`)
d9b96952ec60atagen2026-10-25 09:07 UTC36
  when accessible history ends first, `Cancelled`
d9b96952ec60atagen2026-10-25 09:07 UTC37
- `Paused` with a `failure`, and the gap row goes `Failed`
d9b96952ec60atagen2026-10-25 09:07 UTC38
06c73391cbaeatagen2026-10-25 10:26 UTC39
a gap that is already unavailable answers at once and counts no page. one that
06c73391cbaeatagen2026-10-25 10:26 UTC40
has already joined is gone, so filling it answers `EventNotFound`.
d9b96952ec60atagen2026-10-25 09:07 UTC41
d9b96952ec60atagen2026-10-25 09:07 UTC42
## one operation per room
d9b96952ec60atagen2026-10-25 09:07 UTC43
d9b96952ec60atagen2026-10-25 09:07 UTC44
a room runs one history operation at a time. `RetryRecent` joins a recent load
d9b96952ec60atagen2026-10-25 09:07 UTC45
that is already running. everything else that overlaps, a fill during a recent
d9b96952ec60atagen2026-10-25 09:07 UTC46
load, a recent load during a fill, a second fill, answers `RoomBusy`.
d9b96952ec60atagen2026-10-25 09:07 UTC47
d9b96952ec60atagen2026-10-25 09:07 UTC48
`HistoryRepairCommand::Cancel` stops whatever is running. its own answer is
d9b96952ec60atagen2026-10-25 09:07 UTC49
`Ok`, the cancelled command answers `Interrupted`, and the published outcome is
d9b96952ec60atagen2026-10-25 09:07 UTC50
`Cancelled`, never a failure. pages already committed stay.
d9b96952ec60atagen2026-10-25 09:07 UTC51
d9b96952ec60atagen2026-10-25 09:07 UTC52
## the timeline
d9b96952ec60atagen2026-10-25 09:07 UTC53
d9b96952ec60atagen2026-10-25 09:07 UTC54
gaps appear in `TimelineState.items` as `TimelineItem::HistoryGap`, between the
d9b96952ec60atagen2026-10-25 09:07 UTC55
sections they separate. a row keeps its id while its lifecycle changes, and is
d9b96952ec60atagen2026-10-25 09:07 UTC56
removed once the gap joins. joined gaps never appear.
d9b96952ec60atagen2026-10-25 09:07 UTC57
06c73391cbaeatagen2026-10-25 10:26 UTC58
history lands as incremental `TimelineChanged` diffs, never a `TimelineReset`.
06c73391cbaeatagen2026-10-25 10:26 UTC59
before answering, the room fences on `Timeline::flush_history`, which applies
06c73391cbaeatagen2026-10-25 10:26 UTC60
what the cache already queued without replacing the timeline, so receipts, the
06c73391cbaeatagen2026-10-25 10:26 UTC61
read marker and row ids survive.
d9b96952ec60atagen2026-10-25 09:07 UTC62
d9b96952ec60atagen2026-10-25 09:07 UTC63
a command's answer comes after its outcome is published, and after the mirror
d9b96952ec60atagen2026-10-25 09:07 UTC64
has applied it when one is attached.
d9b96952ec60atagen2026-10-25 09:07 UTC65
d9b96952ec60atagen2026-10-25 09:07 UTC66
## known holes
d9b96952ec60atagen2026-10-25 09:07 UTC67
d9b96952ec60atagen2026-10-25 09:07 UTC68
a limited sync for a room with no events in memory records a plain gap instead
d9b96952ec60atagen2026-10-25 09:07 UTC69
of a history gap (review 2026-10-25 S1), so an interval can still render as
d9b96952ec60atagen2026-10-25 09:07 UTC70
continuous. both `prev_batch` drops in the sdk's `handle_sync` remain.
d9b96952ec60atagen2026-10-25 09:07 UTC71
d9b96952ec60atagen2026-10-25 09:07 UTC72
## wire shapes
d9b96952ec60atagen2026-10-25 09:07 UTC73
d9b96952ec60atagen2026-10-25 09:07 UTC74
serde enums are externally tagged.
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC75
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC76
```json
d9b96952ec60atagen2026-10-25 09:07 UTC77
{ "HistoryRepair": { "FillGap": { "room_id": "!room:example.org", "gap_id": "opaque" } } }
d9b96952ec60atagen2026-10-25 09:07 UTC78
{ "HistoryRepair": { "RetryRecent": { "room_id": "!room:example.org" } } }
d9b96952ec60atagen2026-10-25 09:07 UTC79
{ "HistoryRepair": { "Cancel": { "room_id": "!room:example.org" } } }
9eabb6b7587datagen2026-10-25 07:08 UTC80
```
9eabb6b7587datagen2026-10-25 07:08 UTC81
d9b96952ec60atagen2026-10-25 09:07 UTC82
a gap row.
9eabb6b7587datagen2026-10-25 07:08 UTC83
9eabb6b7587datagen2026-10-25 07:08 UTC84
```json
9eabb6b7587datagen2026-10-25 07:08 UTC85
{
9eabb6b7587datagen2026-10-25 07:08 UTC86
  "HistoryGap": {
d9b96952ec60atagen2026-10-25 09:07 UTC87
    "id": "opaque",
9eabb6b7587datagen2026-10-25 07:08 UTC88
    "older_boundary": "$older:example.org",
9eabb6b7587datagen2026-10-25 07:08 UTC89
    "newer_boundary": "$newer:example.org",
9eabb6b7587datagen2026-10-25 07:08 UTC90
    "lifecycle": "Idle",
9eabb6b7587datagen2026-10-25 07:08 UTC91
    "failure": null
9eabb6b7587datagen2026-10-25 07:08 UTC92
  }
9eabb6b7587datagen2026-10-25 07:08 UTC93
}
9eabb6b7587datagen2026-10-25 07:08 UTC94
```
9eabb6b7587datagen2026-10-25 07:08 UTC95
d9b96952ec60atagen2026-10-25 09:07 UTC96
lifecycle is `Idle`, `Paused`, `Failed` or `Unavailable`. `failure` is set with
d9b96952ec60atagen2026-10-25 09:07 UTC97
`Failed`.
d9b96952ec60atagen2026-10-25 09:07 UTC98
d9b96952ec60atagen2026-10-25 09:07 UTC99
a settled fill.
9eabb6b7587datagen2026-10-25 07:08 UTC100
9eabb6b7587datagen2026-10-25 07:08 UTC101
```json
9eabb6b7587datagen2026-10-25 07:08 UTC102
{
d9b96952ec60atagen2026-10-25 09:07 UTC103
  "room_id": "!room:example.org",
d9b96952ec60atagen2026-10-25 09:07 UTC104
  "gap_id": "opaque",
d9b96952ec60atagen2026-10-25 09:07 UTC105
  "pages": 2,
d9b96952ec60atagen2026-10-25 09:07 UTC106
  "events": 63,
d9b96952ec60atagen2026-10-25 09:07 UTC107
  "phase": "Ready",
d9b96952ec60atagen2026-10-25 09:07 UTC108
  "continuity": "Verified",
d9b96952ec60atagen2026-10-25 09:07 UTC109
  "stop": null,
d9b96952ec60atagen2026-10-25 09:07 UTC110
  "failure": null
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC111
}
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC112
```
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC113
d9b96952ec60atagen2026-10-25 09:07 UTC114
the mirror keeps `State.recent_history`, a room without an entry being idle,
d9b96952ec60atagen2026-10-25 09:07 UTC115
and `State.repairs`, the last fill report per room. closing a room drops both.