atagen

macltok

public
Star0
Watch0
0dfdd69718302c1d2503a83a5efbcec10b9da6c2
3
ssh://brewery@pls.lobotomise.me:33/atagen/macltok.git

Blame: docs/segmented-history.md

d9b96952ec60atagen2026-10-25 09:07 UTC1
# segmented history
d9b96952ec60atagen2026-10-25 09:07 UTC2
d9b96952ec60atagen2026-10-25 09:07 UTC3
a room's history is sections of events with durable gaps between them. the sdk
d9b96952ec60atagen2026-10-25 09:07 UTC4
event cache keeps the sections, gap identity, boundaries and each gap's cursor,
d9b96952ec60atagen2026-10-25 09:07 UTC5
encrypted at rest, across room close and restart. macltok shows gaps inline and
d9b96952ec60atagen2026-10-25 09:07 UTC6
fills them on request.
d9b96952ec60atagen2026-10-25 09:07 UTC7
d9b96952ec60atagen2026-10-25 09:07 UTC8
## opening a room
d9b96952ec60atagen2026-10-25 09:07 UTC9
d9b96952ec60atagen2026-10-25 09:07 UTC10
opening loads retained history and its gaps, then starts a recent load in the
d9b96952ec60atagen2026-10-25 09:07 UTC11
background, so opening never waits on the network. the recent load fetches up
d9b96952ec60atagen2026-10-25 09:07 UTC12
to four pages of 50 events from a fresh server position. if it reaches the
d9b96952ec60atagen2026-10-25 09:07 UTC13
newest retained event the sections join. if it doesn't, a gap is left between
d9b96952ec60atagen2026-10-25 09:07 UTC14
the fresh window and what was retained.
d9b96952ec60atagen2026-10-25 09:07 UTC15
d9b96952ec60atagen2026-10-25 09:07 UTC16
`Event::RecentHistoryChanged` reports it, one event per edge.
d9b96952ec60atagen2026-10-25 09:07 UTC17
d9b96952ec60atagen2026-10-25 09:07 UTC18
- `Loading` as it starts
d9b96952ec60atagen2026-10-25 09:07 UTC19
- `Loaded`, `Cancelled` or `Failed(message)` as it settles
d9b96952ec60atagen2026-10-25 09:07 UTC20
- `Idle` when the room closes, whatever was happening
d9b96952ec60atagen2026-10-25 09:07 UTC21
002c79e4c6ecatagen2026-10-25 10:42 UTC22
`Failed` stays until `HistoryRepairCommand::RetryRecent`, and closing the room
002c79e4c6ecatagen2026-10-25 10:42 UTC23
turns it to `Idle` like anything else. a failed load on a room timeline with no
002c79e4c6ecatagen2026-10-25 10:42 UTC24
events yet falls back to paginating `EngineConfig::initial_history_limit`
002c79e4c6ecatagen2026-10-25 10:42 UTC25
events, so the room isn't left empty.
d9b96952ec60atagen2026-10-25 09:07 UTC26
d9b96952ec60atagen2026-10-25 09:07 UTC27
## filling a gap
d9b96952ec60atagen2026-10-25 09:07 UTC28
d9b96952ec60atagen2026-10-25 09:07 UTC29
`HistoryRepairCommand::FillGap { room_id, gap_id }` walks one gap backwards, up
d9b96952ec60atagen2026-10-25 09:07 UTC30
to four pages of 50, each page committing its events and the gap's new cursor
d9b96952ec60atagen2026-10-25 09:07 UTC31
together. `Event::HistoryRepairChanged` carries a `HistoryRepairReport`, once
d9b96952ec60atagen2026-10-25 09:07 UTC32
with phase `Bridging` as the fill starts and once more as it settles.
d9b96952ec60atagen2026-10-25 09:07 UTC33
d9b96952ec60atagen2026-10-25 09:07 UTC34
- `Ready`, continuity `Verified`. the older section was reached and the gap is gone
d9b96952ec60atagen2026-10-25 09:07 UTC35
- `Paused` with a stop. `Budget` after four pages, `CursorStalled` when the
d9b96952ec60atagen2026-10-25 09:07 UTC36
  server repeats a cursor, `HistoryUnavailable` (continuity `AccessibleStart`)
d9b96952ec60atagen2026-10-25 09:07 UTC37
  when accessible history ends first, `Cancelled`
d9b96952ec60atagen2026-10-25 09:07 UTC38
- `Paused` with a `failure`, and the gap row goes `Failed`
d9b96952ec60atagen2026-10-25 09:07 UTC39
06c73391cbaeatagen2026-10-25 10:26 UTC40
a gap that is already unavailable answers at once and counts no page. one that
06c73391cbaeatagen2026-10-25 10:26 UTC41
has already joined is gone, so filling it answers `EventNotFound`.
d9b96952ec60atagen2026-10-25 09:07 UTC42
d9b96952ec60atagen2026-10-25 09:07 UTC43
## one operation per room
d9b96952ec60atagen2026-10-25 09:07 UTC44
d9b96952ec60atagen2026-10-25 09:07 UTC45
a room runs one history operation at a time. `RetryRecent` joins a recent load
d9b96952ec60atagen2026-10-25 09:07 UTC46
that is already running. everything else that overlaps, a fill during a recent
d9b96952ec60atagen2026-10-25 09:07 UTC47
load, a recent load during a fill, a second fill, answers `RoomBusy`.
d9b96952ec60atagen2026-10-25 09:07 UTC48
d9b96952ec60atagen2026-10-25 09:07 UTC49
`HistoryRepairCommand::Cancel` stops whatever is running. its own answer is
d9b96952ec60atagen2026-10-25 09:07 UTC50
`Ok`, the cancelled command answers `Interrupted`, and the published outcome is
d9b96952ec60atagen2026-10-25 09:07 UTC51
`Cancelled`, never a failure. pages already committed stay.
d9b96952ec60atagen2026-10-25 09:07 UTC52
d9b96952ec60atagen2026-10-25 09:07 UTC53
## the timeline
d9b96952ec60atagen2026-10-25 09:07 UTC54
d9b96952ec60atagen2026-10-25 09:07 UTC55
gaps appear in `TimelineState.items` as `TimelineItem::HistoryGap`, between the
d9b96952ec60atagen2026-10-25 09:07 UTC56
sections they separate. a row keeps its id while its lifecycle changes, and is
d9b96952ec60atagen2026-10-25 09:07 UTC57
removed once the gap joins. joined gaps never appear.
d9b96952ec60atagen2026-10-25 09:07 UTC58
002c79e4c6ecatagen2026-10-25 10:42 UTC59
history lands as `TimelineChanged` diffs, never a `TimelineReset`. they are
002c79e4c6ecatagen2026-10-25 10:42 UTC60
incremental unless the cache's loaded window changed under the operation, after
002c79e4c6ecatagen2026-10-25 10:42 UTC61
a restart or a shrink, when the sdk sends the room as one `Reset` diff. before
002c79e4c6ecatagen2026-10-25 10:42 UTC62
answering, the room fences on `Timeline::flush_history`, which applies what the
002c79e4c6ecatagen2026-10-25 10:42 UTC63
cache already queued without replacing the timeline, so receipts, the read
002c79e4c6ecatagen2026-10-25 10:42 UTC64
marker and row ids survive. a failed or cancelled operation that committed
002c79e4c6ecatagen2026-10-25 10:42 UTC65
pages is fenced the same way.
d9b96952ec60atagen2026-10-25 09:07 UTC66
d9b96952ec60atagen2026-10-25 09:07 UTC67
a command's answer comes after its outcome is published, and after the mirror
d9b96952ec60atagen2026-10-25 09:07 UTC68
has applied it when one is attached.
d9b96952ec60atagen2026-10-25 09:07 UTC69
d9b96952ec60atagen2026-10-25 09:07 UTC70
## wire shapes
d9b96952ec60atagen2026-10-25 09:07 UTC71
d9b96952ec60atagen2026-10-25 09:07 UTC72
serde enums are externally tagged.
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC73
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC74
```json
d9b96952ec60atagen2026-10-25 09:07 UTC75
{ "HistoryRepair": { "FillGap": { "room_id": "!room:example.org", "gap_id": "opaque" } } }
d9b96952ec60atagen2026-10-25 09:07 UTC76
{ "HistoryRepair": { "RetryRecent": { "room_id": "!room:example.org" } } }
d9b96952ec60atagen2026-10-25 09:07 UTC77
{ "HistoryRepair": { "Cancel": { "room_id": "!room:example.org" } } }
9eabb6b7587datagen2026-10-25 07:08 UTC78
```
9eabb6b7587datagen2026-10-25 07:08 UTC79
d9b96952ec60atagen2026-10-25 09:07 UTC80
a gap row.
9eabb6b7587datagen2026-10-25 07:08 UTC81
9eabb6b7587datagen2026-10-25 07:08 UTC82
```json
9eabb6b7587datagen2026-10-25 07:08 UTC83
{
9eabb6b7587datagen2026-10-25 07:08 UTC84
  "HistoryGap": {
d9b96952ec60atagen2026-10-25 09:07 UTC85
    "id": "opaque",
9eabb6b7587datagen2026-10-25 07:08 UTC86
    "older_boundary": "$older:example.org",
9eabb6b7587datagen2026-10-25 07:08 UTC87
    "newer_boundary": "$newer:example.org",
9eabb6b7587datagen2026-10-25 07:08 UTC88
    "lifecycle": "Idle",
9eabb6b7587datagen2026-10-25 07:08 UTC89
    "failure": null
9eabb6b7587datagen2026-10-25 07:08 UTC90
  }
9eabb6b7587datagen2026-10-25 07:08 UTC91
}
9eabb6b7587datagen2026-10-25 07:08 UTC92
```
9eabb6b7587datagen2026-10-25 07:08 UTC93
d9b96952ec60atagen2026-10-25 09:07 UTC94
lifecycle is `Idle`, `Paused`, `Failed` or `Unavailable`. `failure` is set with
d9b96952ec60atagen2026-10-25 09:07 UTC95
`Failed`.
d9b96952ec60atagen2026-10-25 09:07 UTC96
d9b96952ec60atagen2026-10-25 09:07 UTC97
a settled fill.
9eabb6b7587datagen2026-10-25 07:08 UTC98
9eabb6b7587datagen2026-10-25 07:08 UTC99
```json
9eabb6b7587datagen2026-10-25 07:08 UTC100
{
d9b96952ec60atagen2026-10-25 09:07 UTC101
  "room_id": "!room:example.org",
d9b96952ec60atagen2026-10-25 09:07 UTC102
  "gap_id": "opaque",
d9b96952ec60atagen2026-10-25 09:07 UTC103
  "pages": 2,
d9b96952ec60atagen2026-10-25 09:07 UTC104
  "events": 63,
d9b96952ec60atagen2026-10-25 09:07 UTC105
  "phase": "Ready",
d9b96952ec60atagen2026-10-25 09:07 UTC106
  "continuity": "Verified",
d9b96952ec60atagen2026-10-25 09:07 UTC107
  "stop": null,
d9b96952ec60atagen2026-10-25 09:07 UTC108
  "failure": null
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC109
}
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC110
```
c9f0fd0ee1ccatagen2026-10-25 06:16 UTC111
d9b96952ec60atagen2026-10-25 09:07 UTC112
the mirror keeps `State.recent_history`, a room without an entry being idle,
d9b96952ec60atagen2026-10-25 09:07 UTC113
and `State.repairs`, the last fill report per room. closing a room drops both.