macltok

# Persistent mentions discovery

This records the SDK audit and current persistent mentions implementation.

Persistent sessions now commit discovery with Sliding Sync responses, retain an

encrypted inbox, and expose it through `ClientHandle` and the mirror.

Historical coverage remains incomplete because room-list sync and notification

history cannot prove that every past mention was observed.

The audited dependency is Matrix Rust SDK 0.18.0 at fork revision

`cc2b13ea185c0f694523a3a48d8aadfe366dd767`. macltok now pins fork revision

`7aace883fbe03d4c3c118b55188fe9519ba8ce48`, which adds the response commit

hook described below. SDK paths below are relative to the audited revision.

## Findings

The SDK can deliver and decrypt events without opening a UI timeline. macltok

records observed events and recovers candidates through notification history,

but neither source establishes complete historical coverage.

The SDK audit found that an awaited event handler can hold back the Sliding

Sync position until a macltok transaction commits, while a background

broadcast consumer cannot provide that guarantee. Handlers cannot propagate

commit errors, and response processing outlives cancellation of the calling

sync task. These limits motivated the fallible response-commit hook now

implemented in the pinned SDK fork.

## Sync coverage

macltok constructs `SyncService` in `src/engine/mod.rs` and uses the SDK room

list configuration. The list starts with the selective range `0..=19`. After

setup it grows by 100 rooms per response, without an SDK-side maximum, until

it reaches the server's room count. Recovery and sufficiently long idle

periods can return it to the initial selective range before it grows again.

The default list timeline limit is one event per room on initial delivery.

Subsequent changes for covered rooms continue arriving through sync.

Opening a room separately subscribes it with a

timeline limit of 20. macltok's `update_room_subscriptions` and

`watchers::room_subscriptions` manage those explicit subscriptions for active

room actors only. The mentions response hook filters entries to joined rooms.

The built-in list still includes invited rooms.

Sources are `crates/matrix-sdk-ui/src/room_list_service/mod.rs`,

`crates/matrix-sdk-ui/src/room_list_service/state.rs`, and

`crates/matrix-sdk/src/sliding_sync/list/request_generator.rs`.

The default one-event initial window cannot recover all earlier pings, including

messages sent while the client was offline. Raising

`SyncServiceBuilder::with_room_list_timeline_limit` increases the initial

window but does not establish continuous history. The response hook processes

joined-room events and records limited timelines of rooms it had already

observed, checkpoint discontinuities, and recovery gaps. A room seen for the

first time adds no gap, since every room starts that way.

## Observation and durable commit

`Client::subscribe_to_all_room_updates` returns a bounded broadcast receiver

of `RoomUpdates`. Its `joined` map includes processed timeline events with raw

decrypted content when available and optional evaluated push actions. It does

not require a UI timeline. Lag is an error that requires recovery, not an

invitation to skip to the next update.

`Client::observe_events` retains only the latest observation and can miss

intermediate events. It is unsuitable for durable discovery.

An earlier probe registers `Client::add_event_handler` for

`Raw<AnySyncTimelineEvent>` with `Room` context before sync starts. Its awaited

handler demonstrates event ordering, but it cannot report commit failures

reliably and is not the production ingestion path. Production uses the

fallible Sliding Sync response hook described below.

For Sliding Sync v5, the response processor awaits event handlers before

updating and persisting `pos`. The response hook commits the inbox first, so

a crash permits replay without skipping committed candidates. Stable event

keys make replay idempotent. The hook acknowledgement does not wait for

context capture, a UI subscriber, or the engine event stream to drain.

For ordinary event handlers, the SDK logs and discards handler errors, so

returning an error after a failed write still permits `pos` to advance. A

pending handler alone also does not provide a reliable shutdown path. The SDK

runs response handling in a detached task that outlives cancellation of

`sync_once` or its caller. Aborting the engine's sync task cannot prevent a

suspended handler from later returning and advancing the position. Registering

a handler and placing work on a queue without awaiting its commit does not

close the crash window.

macltok uses shared `pos` through its SDK service and commits each response

batch through one serialised mentions actor before the hook acknowledges it.

The actor records events from joined rooms and from rooms the same response

leaves, raw encrypted candidates, limited timeline gaps, and the connection

checkpoint in one store commit. Stable room and event keys make replay

idempotent. The mentions watch snapshot is replaced before the ack, while

mirror and event-stream delivery and context capture run after it. The classic

sync `next_batch` ordering differs and does not establish this guarantee.

Nor does this ordering make arbitrary concurrent sync loops safe.

The checkpoint keeps the position the committed response started from as well

as the one it ended at. A crash after the commit but before the SDK saves `pos`

replays from that earlier position, and the replay is a superset of what was

committed, so it is not treated as a discontinuity. Any other starting position

records a `SyncDiscontinuity` gap.

The hook never fails sync. A failed write keeps the inbox in memory, records a

`StoreFailed` gap, acks, and retries the write a minute later, so a restart

before a write lands shows up as a discontinuity. The gap goes again once a

later write lands, since nothing was lost. The hook gives up after ten seconds

waiting for room in the actor's queue, drops the response and records

`HookTimedOut`. Once queued, it waits another ten seconds for the commit and

then acks regardless, because a queued response still commits in order.

Sources are `crates/matrix-sdk/src/event_handler/mod.rs`,

`crates/matrix-sdk/src/event_handler/static_events.rs`,

`crates/matrix-sdk/src/sliding_sync/client.rs`,

`crates/matrix-sdk/src/sliding_sync/mod.rs`, and

`crates/matrix-sdk/src/sliding_sync/cache.rs`.

### SDK commit hook implementation

The local SDK workspace at `../matrix-rust-sdk-mentions` implements the

fallible asynchronous response-commit hook through `ResponseCommitHook` and

`ResponseCommitBatch`. `SyncServiceBuilder` forwards it only to its room-list

connection, so independent notification lookups are unaffected.

The hook receives the processed room events, original room payloads, limited

flags, pagination tokens, connection identity, and previous and next position

identifiers. It runs before response handlers and observers and before `pos`

is updated or saved. SDK room state may already have been saved. A successful

acknowledgement follows the macltok inbox transaction. A failed or dropped

acknowledgement returns a sync error without advancing the position.

The hook runs outside SDK store locks and supports local futures on wasm and

sendable futures natively. The default absent hook preserves existing SDK

behaviour. Detached response processing observes a dropped store

acknowledgement after its caller is cancelled. Checkpoint replay works after

SDK room state was saved, and stable event keys deduplicate repeated commits.

The SDK audit and focused validation covered commit ordering, failed and

dropped acknowledgements, caller cancellation, replay after room-state

persistence, restoring a previous checkpoint, absent-hook behaviour, builder

forwarding, and connection isolation. The SDK UI library also received a

focused wasm compilation check with its `js` feature. The SDK changes are

published to the user's fork under `mentions-commit-hook`, and the manifest

and both macltok lockfiles pin revision

`7aace883fbe03d4c3c118b55188fe9519ba8ce48`.

## Event cache and delayed decryption

`EventCache::subscribe` starts global room-update ingestion. The SDK room list

service already calls it during construction. Its generic update subscription

identifies the affected room, then `Room::event_cache` and

`RoomEventCache::find_event` can retrieve an event. This does not require a

`matrix-sdk-ui::Timeline` or a retained subscriber per room.

`EventCache::subscribe_to_decryption_reports` reports resolved event IDs,

lag, and backup availability. `EventCache::request_decryption` accepts room

and Megolm session IDs for explicit retries. `Room::decrypt_event` is also

public and can decrypt a retained raw event independently of a UI timeline.

The late-key probe confirms automatic decryption and retrieval of a structured

mention through the global cache without a UI timeline. Cached push actions

are optional. Re-evaluate them with `Room::event_push_actions` when needed,

and treat missing push context as unresolved rather than a negative result.

The room's own membership state is needed to build that push context.

The cache is not a durable ingestion journal. Its room broadcasts and generic

updates can lag, and cache retention can remove events. SQLite and IndexedDB

builders select persistent SDK event-cache stores, while the default

store-free client uses memory. Even with persistence, the asynchronous cache

consumer is not awaited before Sliding Sync saves its position.

The inbox retains unresolved encrypted candidates as encrypted raw events

within a configurable bound. It retries due candidates at startup. A report of

resolved events retries the encrypted candidates of that room, and lag or a

new backup retries all of them. Candidates still unresolved a week after they

were first seen expire behind a `CandidateExpired` gap. Over the bound the

oldest events go first, and eviction sets the coverage truncation flag. Imported keys and backups cannot reconstruct events

already discarded by every cache.

Sources are `crates/matrix-sdk/src/event_cache/mod.rs`,

`crates/matrix-sdk/src/event_cache/redecryptor.rs`,

`crates/matrix-sdk/src/event_cache/caches/room/mod.rs`, and

`crates/matrix-sdk/src/room/mod.rs`.

## Mention policy

Structured `m.mentions.user_ids` and `m.mentions.room` supply semantic

candidates. The classifier does not infer pings from rendered bodies or

unread counts. A reply relationship alone does not prove that a reply notifies

the user, so reply reasons require confirmation that the referenced event

belongs to the signed-in user plus either an explicit user mention or a

notifying highlight.

`Room::event_push_actions` evaluates push rules but does not independently

filter ignored senders. The service reads the ignore list once per response

and again when it changes, and rechecks entries then.

A room mute can suppress evaluated notification actions while structured

mention metadata remains present. The classifier retains explicit user

mentions in muted rooms and marks notification delivery as suppressed. Room-wide

reasons require the relevant push conditions and sender power level. Missing

authorisation context remains unresolved for reevaluation. An event that

notifies and highlights with no other reason, such as a loud keyword, enters the

inbox as `Keyword`. The SDK's keyword rules only notify and play a sound, so

`add_notification_keyword` reads the content rule the SDK names after the

keyword and adds a highlight tweak when it lacks one, leaving its other actions

alone. A rule the user made quiet stays quiet apart from the highlight. A

change of reason alone keeps an entry's saved context. The classifier excludes the signed-in

user's own events even when they contain explicit mention metadata.

The existing `NotificationClient` combines notification evaluation with

ignored-sender filtering. Its public lookup takes known room and event IDs.

It is useful for resolving a candidate, but filtered results erase the

distinction between mute policy and an absent semantic mention. It is not the

sole classifier for this inbox.

## Historical candidates and gaps

Ruma exposes

`api::client::push::get_notifications::v3::Request`, sent with `Client::send`.

It accepts opaque `from`, optional `limit`, and optional `only` fields.

Responses contain notifications and an optional `next_token`. macltok requests

50 notifications per page. Entries include raw events, room IDs, timestamps,

actions, and server notification read state, which does not initialise or

replace local inbox read state.

The service uses this endpoint as a bounded candidate source, validates

events after decryption, and applies the live mention policy. Each page's

observed candidates and cursor are committed together in the encrypted inbox,

including unresolved encrypted candidates. A failed commit leaves the prior

cursor available for replay.

An exhausted notification list does not mean complete room history. Muted-room

pings, encrypted mentions not recognised by the server, expired notifications,

and server-specific retention can be absent. Recovery inspects at most 20 pages

of 50 notifications per run and resumes from committed cursors later. The

older-history cursor is separate from the cursor for scans from the newest

notifications. A newest scan records an anchor and runs at startup and every

60 seconds. It follows the committed cursor until it reaches that anchor,

resuming on later runs when needed to find newly added notifications.

SDK cache gaps retain pagination tokens and support room pagination, but

running that pagination mutates the live cache. Mention context capture uses

independent `/context` and `/messages` requests. Tokens may expire. Context can

be retried around a known event while the room remains joined, and known gaps

remain visible when continuity cannot be established.

Coverage exposes the earliest observed timestamp, history-endpoint support,

recovery activity, known room gaps, unresolved candidate count, and retention

truncation. The earliest timestamp is a summary, not proof that every room is

covered after that time. Each gap carries a typed `MentionGapReason`.

Complete offline mention recovery across arbitrary muted and encrypted rooms

is not supplied by this endpoint or by the current sync configuration. It

would require a retained, replayable event source or bounded per-room history

recovery with proof of continuity. The notification endpoint does not provide

that guarantee.

## Storage, context, and API

`src/mentions/store` implements encrypted SQLite and IndexedDB record stores.

Persistent session construction opens the account store, and local logout

deletes it alongside the SDK stores. The mentions actor owns the store for

inbox commits and discovery checkpoints.

The passphrase wraps a random SDK `StoreCipher`. Logical keys are hashed, and

authenticated encrypted records include schema, account identity, and logical

key. Restore treats mismatched accounts, older schemas, corrupt records, and

inconsistent record counts or byte totals as unreadable, empties the store and

records a `StoreReset` gap rather than failing `Account::build`. A busy or

failing store and one written by a newer macltok are left alone, and the inbox

runs in memory behind `StoreFailed`. The busy or failing case is retried every

minute, and when it opens, what it held and what was built in memory since are

merged.

Atomic transactions compare the previous encrypted header before publishing a

new revision, so stale writers cannot overwrite a newer checkpoint.

The default budget is 64 MiB of encrypted records, hashed keys, header, and

wrapped cipher, excluding database page overhead. The store record cap is

35,128. The inbox defaults to 5,000 entries, 25,000 shared context events, and

5,000 unresolved candidates within that byte budget. When a size or record

limit requires eviction, the store drops bounded data and marks coverage

truncated.

The storage tests cover atomic restore of records and checkpoints, stale

writers, capacity failure, encrypted physical records, resets on wrong

credentials or account, corruption and unsupported schemas, and injected

SQLite write failure. The inbox tests cover failed writes, checkpoint-only

writes, replay from the previous checkpoint, reset of an unreadable inbox,

notification recovery, live context updates, and capture persistence across

session resume.

The store holds three records. `inbox` carries entries, saved events, context

windows, candidates, edits, redactions, coverage and recovery cursors. `sync`

carries the connection checkpoints and the inbox revision it belongs to.

`rooms` carries the rooms already observed, which drop out when left. Every

response writes `sync`, and `inbox` or `rooms` only when the response changed

them, all in one commit. Whether the inbox changed is decided by comparing the

rooms the response touched, so a quiet response costs those rooms and one small

record, while a changed inbox is still reserialised and re-encrypted whole, in

one serialisation pass. Background completions are applied in memory and

written together at most once a second, and published on that tick even while

writes back off. When the budget is short, eviction runs against a size

estimate and the inbox is written once.

Decrypted previews, saved context, raw candidates, pending capture state,

media source registrations, read state, coverage, and sync checkpoints stay

inside the authenticated encrypted payloads. SDK SQLite custom values remain plaintext and are not used

for decrypted mentions.

Context capture in `src/engine/context.rs` sizes the before and after requests

independently and retains the media registrations produced by normalisation.

Each capture requests up to 10 events on each side. At most two capture jobs

run concurrently, with up to 128 pending or failed entries considered for

scheduling at a time. Capture uses independent `/context` and `/messages`

requests, so it does not paginate the live cache. Media source registrations

are encrypted with saved events and registered again when context is read.

Media bodies and key material do not enter the mentions public model.

One account-scoped mentions actor serialises store commits for entries,

unresolved candidates, shared context events, read state, revisions, discovery

checkpoints, and retention decisions.

Pending capture state is durable with the inbox, so interrupted capture can

resume after startup. When a room leaves the joined state, saved context

remains readable and its capture state becomes unavailable. Refetch is

disabled after departure.

Store publication follows commit. `ClientHandle` exposes page, context, retry,

read, read-through, and recovery methods. The public `Command`, mention models,

mirror state, and updates support serde when the `serde` feature is enabled,

so native and browser consumers can transport them. Browser worker integration

is a consumer phase after SDK and macltok revision publication. Malice's worker

still needs mention variants in its explicit `Request` dispatch and TypeScript

methods.

Mirror snapshots contain at most 50 entry previews, coverage, unread count,

and a committed revision boundary, with no saved context bodies. The watcher

coalesces updates and computes upserts and removals relative to the previous

bounded snapshot it observed. A missed update therefore leaves the latest

snapshot authoritative. A removal can mean an entry fell out of the top 50,

and does not imply that it was deleted from the inbox. Pagination uses

timestamp and stable event key within a discovery revision boundary.

Read-through boundaries exclude mentions first discovered or reactivated after

that boundary, including older messages resolved by late decryption.

## Executable probes

`tests/mentions_discovery.rs` exercises unopened-room update and cache access,

push-rule behaviour for ignored senders and muted rooms, and typed

notification-history pagination.

`tests/mentions_decryption.rs` exchanges a real encrypted event and a delayed

room key between two mock clients, then retrieves the decrypted mention

without opening a UI timeline.

`tests/mentions_checkpoint.rs` holds an event handler pending and verifies

that the SDK has not persisted its position. It cancels the calling task,

then acknowledges the handler and verifies that detached response processing

still saves the position. A new Sliding Sync instance restores that position.

This is the historical event-handler ordering probe. It does not exercise the

production response-commit hook or the durable inbox transaction. It also

demonstrates that returning a handler error still advances the position.

`tests/mentions_inbox.rs` covers live mention classification, read-through

with revision boundaries, independent context limits, and persistence across

session resume. `tests/mentions_late_decryption.rs` verifies that a delayed room key

updates the production inbox once. `src/mentions/service_tests.rs` covers

failed and checkpoint-only writes, checkpoint replay, store reset,

notification-history recovery, and context updates.

`src/mentions/store/tests.rs` exercises encrypted record restore, capacity,

stale writers, and corruption handling.

These probes establish SDK behaviour against mocked responses. They do not

establish homeserver notification retention, room-list completeness, or

complete history on a deployed server.

Focused validation passed 63 native tests and 12 shared inbox and IndexedDB

tests in headless Firefox. Compilation checks also pass without storage

features and for wasm with IndexedDB and mirror support. The full flake gate

was skipped by request.