macltok

# Persistent mentions discovery

This records the SDK audit and current persistent mentions implementation.

Persistent sessions now commit discovery with Sliding Sync responses, retain an

encrypted inbox, and expose it through `ClientHandle` and the mirror.

Historical coverage remains incomplete because room-list sync and notification

history cannot prove that every past mention was observed.

The audited dependency is Matrix Rust SDK 0.18.0 at fork revision

`cc2b13ea185c0f694523a3a48d8aadfe366dd767`. macltok now pins fork revision

`7aace883fbe03d4c3c118b55188fe9519ba8ce48`, which adds the response commit

hook described below. SDK paths below are relative to the audited revision.

## Findings

The SDK can deliver and decrypt events without opening a UI timeline. macltok

records observed events and recovers candidates through notification history,

but neither source establishes complete historical coverage.

The SDK audit found that an awaited event handler can hold back the Sliding

Sync position until a macltok transaction commits, while a background

broadcast consumer cannot provide that guarantee. Handlers cannot propagate

commit errors, and response processing outlives cancellation of the calling

sync task. These limits motivated the fallible response-commit hook now

implemented in the pinned SDK fork.

## Sync coverage

macltok constructs `SyncService` in `src/engine/mod.rs` and uses the SDK room

list configuration. The list starts with the selective range `0..=19`. After

setup it grows by 100 rooms per response, without an SDK-side maximum, until

it reaches the server's room count. Recovery and sufficiently long idle

periods can return it to the initial selective range before it grows again.

The default list timeline limit is one event per room on initial delivery.

Subsequent changes for covered rooms continue arriving through sync.

Opening a room separately subscribes it with a

timeline limit of 20. macltok's `update_room_subscriptions` and

`watchers::room_subscriptions` manage those explicit subscriptions for active

room actors only. The mentions response hook filters entries to joined rooms.

The built-in list still includes invited rooms.

Sources are `crates/matrix-sdk-ui/src/room_list_service/mod.rs`,

`crates/matrix-sdk-ui/src/room_list_service/state.rs`, and

`crates/matrix-sdk/src/sliding_sync/list/request_generator.rs`.

The default one-event initial window cannot recover all earlier pings, including

messages sent while the client was offline. Raising

`SyncServiceBuilder::with_room_list_timeline_limit` increases the initial

window but does not establish continuous history. The response hook processes

joined-room events and records limited timelines, initial room coverage,

checkpoint discontinuities, and recovery gaps.

## Observation and durable commit

`Client::subscribe_to_all_room_updates` returns a bounded broadcast receiver

of `RoomUpdates`. Its `joined` map includes processed timeline events with raw

decrypted content when available and optional evaluated push actions. It does

not require a UI timeline. Lag is an error that requires recovery, not an

invitation to skip to the next update.

`Client::observe_events` retains only the latest observation and can miss

intermediate events. It is unsuitable for durable discovery.

An earlier probe registers `Client::add_event_handler` for

`Raw<AnySyncTimelineEvent>` with `Room` context before sync starts. Its awaited

handler demonstrates event ordering, but it cannot report commit failures

reliably and is not the production ingestion path. Production uses the

fallible Sliding Sync response hook described below.

For Sliding Sync v5, the response processor awaits event handlers before

updating and persisting `pos`. The response hook commits the inbox first, so

a crash permits replay without skipping committed candidates. Stable event

keys make replay idempotent. The hook acknowledgement does not wait for

context capture, a UI subscriber, or the engine event stream to drain.

For ordinary event handlers, the SDK logs and discards handler errors, so

returning an error after a failed write still permits `pos` to advance. A

pending handler alone also does not provide a reliable shutdown path. The SDK

runs response handling in a detached task that outlives cancellation of

`sync_once` or its caller. Aborting the engine's sync task cannot prevent a

suspended handler from later returning and advancing the position. Registering

a handler and placing work on a queue without awaiting its commit does not

close the crash window.

macltok uses shared `pos` through its SDK service and commits each response

batch through one serialized mentions actor before the hook acknowledges it.

The actor records joined-room events, raw encrypted candidates, limited

timeline gaps, and the connection checkpoint in the same inbox revision.

Stable room and event keys make replay idempotent. Context capture, mirror

publication, and event-stream delivery continue after the commit. The classic

sync `next_batch` ordering differs and does not establish this guarantee.

Nor does this ordering make arbitrary concurrent sync loops safe.

Sources are `crates/matrix-sdk/src/event_handler/mod.rs`,

`crates/matrix-sdk/src/event_handler/static_events.rs`,

`crates/matrix-sdk/src/sliding_sync/client.rs`,

`crates/matrix-sdk/src/sliding_sync/mod.rs`, and

`crates/matrix-sdk/src/sliding_sync/cache.rs`.

### SDK commit hook implementation

The local SDK workspace at `../matrix-rust-sdk-mentions` implements the

fallible asynchronous response-commit hook through `ResponseCommitHook` and

`ResponseCommitBatch`. `SyncServiceBuilder` forwards it only to its room-list

connection, so independent notification lookups are unaffected.

The hook receives the processed room events, original room payloads, limited

flags, pagination tokens, connection identity, and previous and next position

identifiers. It runs before response handlers and observers and before `pos`

is updated or saved. SDK room state may already have been saved. A successful

acknowledgement follows the macltok inbox transaction. A failed or dropped

acknowledgement returns a sync error without advancing the position.

The hook runs outside SDK store locks and supports local futures on wasm and

sendable futures natively. The default absent hook preserves existing SDK

behavior. Detached response processing observes a dropped store

acknowledgement after its caller is cancelled. Checkpoint replay works after

SDK room state was saved, and stable event keys deduplicate repeated commits.

The SDK audit and focused validation covered commit ordering, failed and

dropped acknowledgements, caller cancellation, replay after room-state

persistence, restoring a previous checkpoint, absent-hook behavior, builder

forwarding, and connection isolation. The SDK UI library also received a

focused wasm compilation check with its `js` feature. The SDK changes are

published to the user's fork under `mentions-commit-hook`, and the manifest

and both macltok lockfiles pin revision

`7aace883fbe03d4c3c118b55188fe9519ba8ce48`.

## Event cache and delayed decryption

`EventCache::subscribe` starts global room-update ingestion. The SDK room list

service already calls it during construction. Its generic update subscription

identifies the affected room, then `Room::event_cache` and

`RoomEventCache::find_event` can retrieve an event. This does not require a

`matrix-sdk-ui::Timeline` or a retained subscriber per room.

`EventCache::subscribe_to_decryption_reports` reports resolved event IDs,

lag, and backup availability. `EventCache::request_decryption` accepts room

and Megolm session IDs for explicit retries. `Room::decrypt_event` is also

public and can decrypt a retained raw event independently of a UI timeline.

The late-key probe confirms automatic decryption and retrieval of a structured

mention through the global cache without a UI timeline. Cached push actions

are optional. Re-evaluate them with `Room::event_push_actions` when needed,

and treat missing push context as unresolved rather than a negative result.

The room's own membership state is needed to build that push context.

The cache is not a durable ingestion journal. Its room broadcasts and generic

updates can lag, and cache retention can remove events. SQLite and IndexedDB

builders select persistent SDK event-cache stores, while the default

store-free client uses memory. Even with persistence, the asynchronous cache

consumer is not awaited before Sliding Sync saves its position.

The inbox retains unresolved encrypted candidates as encrypted raw events

within a configurable bound. It retries due candidates at startup and resets

their retry time when decryption reports arrive. Candidate eviction sets the

coverage truncation flag. Imported keys and backups cannot reconstruct events

already discarded by every cache.

Sources are `crates/matrix-sdk/src/event_cache/mod.rs`,

`crates/matrix-sdk/src/event_cache/redecryptor.rs`,

`crates/matrix-sdk/src/event_cache/caches/room/mod.rs`, and

`crates/matrix-sdk/src/room/mod.rs`.

## Mention policy

Structured `m.mentions.user_ids` and `m.mentions.room` supply semantic

candidates. The classifier does not infer pings from rendered bodies or

unread counts. A reply relationship alone does not prove that a reply notifies

the user, so reply reasons require notification metadata and confirmation that

the referenced event belongs to the signed-in user.

`Room::event_push_actions` evaluates push rules but does not independently

filter ignored senders. The classifier calls `Client::is_user_ignored` and

rechecks entries when the ignore list changes.

A room mute can suppress evaluated notification actions while structured

mention metadata remains present. The classifier retains explicit user

mentions in muted rooms and marks notification delivery as suppressed. Reply

reasons require both a reply to the user and a notifying highlight. Room-wide

reasons require the relevant push conditions and sender power level. Missing

authorization context remains unresolved for reevaluation. Keyword-only

notifications are outside the inbox. The classifier excludes the signed-in

user's own events even when they contain explicit mention metadata.

The existing `NotificationClient` combines notification evaluation with

ignored-sender filtering. Its public lookup takes known room and event IDs.

It is useful for resolving a candidate, but filtered results erase the

distinction between mute policy and an absent semantic mention. It is not the

sole classifier for this inbox.

## Historical candidates and gaps

Ruma exposes

`api::client::push::get_notifications::v3::Request`, sent with `Client::send`.

It accepts opaque `from`, optional `limit`, and optional `only` fields.

Responses contain notifications and an optional `next_token`. macltok requests

50 notifications per page. Entries include raw events, room IDs, timestamps,

actions, and server notification read state, which does not initialize or

replace local inbox read state.

The service uses this endpoint as a bounded candidate source, validates

events after decryption, and applies the live mention policy. Each page's

observed candidates and cursor are committed together in the encrypted inbox,

including unresolved encrypted candidates. A failed commit leaves the prior

cursor available for replay.

An exhausted notification list does not mean complete room history. Muted-room

pings, encrypted mentions not recognized by the server, expired notifications,

and server-specific retention can be absent. Recovery inspects at most 20 pages

of 50 notifications per run and resumes from committed cursors later. The

older-history cursor is separate from the cursor for scans from the newest

notifications. A newest scan records an anchor and runs at startup and every

60 seconds. It follows the committed cursor until it reaches that anchor,

resuming on later runs when needed to find newly added notifications.

SDK cache gaps retain pagination tokens and support room pagination, but

running that pagination mutates the live cache. Mention context capture uses

independent `/context` and `/messages` requests. Tokens may expire. Context can

be retried around a known event while the room remains joined, and known gaps

remain visible when continuity cannot be established.

Coverage exposes the earliest observed timestamp, history-endpoint support,

recovery activity, known room gaps, unresolved candidate count, and retention

truncation. The earliest timestamp is a summary, not proof that every room is

covered after that time. The inbox keeps its incomplete-history flag even

after notification pagination ends.

Complete offline mention recovery across arbitrary muted and encrypted rooms

is not supplied by this endpoint or by the current sync configuration. It

would require a retained, replayable event source or bounded per-room history

recovery with proof of continuity. The notification endpoint does not provide

that guarantee.

## Storage, context, and API

`src/mentions/store` implements encrypted SQLite and IndexedDB record stores.

Persistent session construction opens the account store, and local logout

deletes it alongside the SDK stores. The mentions actor owns the store for

inbox commits and discovery checkpoints.

The passphrase wraps a random SDK `StoreCipher`. Logical keys are hashed, and

authenticated encrypted records include schema, account identity, and logical

key. Restore rejects mismatched accounts, unsupported schemas, corrupt records,

and inconsistent record counts or byte totals. Atomic transactions compare the

previous encrypted header before publishing a new revision, so stale writers

cannot overwrite a newer checkpoint.

The default budget is 64 MiB of encrypted records, hashed keys, header, and

wrapped cipher, excluding database page overhead. The store record cap is

35,128. The inbox defaults to 5,000 entries, 25,000 shared context events, and

5,000 unresolved candidates within that byte budget. When a size or record

limit requires eviction, the store drops bounded data and marks coverage

truncated.

The storage tests cover atomic restore of records and checkpoints, stale

writers, capacity failure, encrypted physical records, wrong credentials or

account, corruption, unsupported schemas, and injected SQLite write failure.

The inbox tests cover batch failure, notification recovery, live context

updates, and capture persistence across session resume.

The inbox is serialized into one `inbox` value and encrypted as a single

record on every revision. Each update reserializes and re-encrypts the retained

inbox, so write cost scales with retained data. Decrypted previews, saved

context, raw candidates, pending capture state, media source registrations,

read state, coverage, and sync checkpoints stay inside the authenticated

encrypted payload. SDK SQLite custom values remain plaintext and are not used

for decrypted mentions.

Context capture in `src/engine/context.rs` sizes the before and after requests

independently and retains the media registrations produced by normalization.

Each capture requests up to 10 events on each side. At most two capture jobs

run concurrently, with up to 128 pending or failed entries considered for

scheduling at a time. Capture uses independent `/context` and `/messages`

requests, so it does not paginate the live cache. Media source registrations

are encrypted with saved events and registered again when context is read.

Media bodies and key material do not enter the mentions public model.

One account-scoped mentions actor serializes store commits for entries,

unresolved candidates, shared context events, read state, revisions, discovery

checkpoints, and retention decisions.

Pending capture state is durable with the inbox, so interrupted capture can

resume after startup. When a room leaves the joined state, saved context

remains readable and its capture state becomes unavailable. Refetch is

disabled after departure.

Store publication follows commit. `ClientHandle` exposes page, context, retry,

read, read-through, and recovery methods. The public `Command`, mention models,

mirror state, and updates support serde when the `serde` feature is enabled,

so native and browser consumers can transport them. Browser worker integration

is a consumer phase after SDK and macltok revision publication. Malice's worker

still needs mention variants in its explicit `Request` dispatch and TypeScript

methods.

Mirror snapshots contain at most 50 entry previews, coverage, unread count,

and a committed revision boundary, with no saved context bodies. The watcher

coalesces updates and computes upserts and removals relative to the previous

bounded snapshot it observed. A missed update therefore leaves the latest

snapshot authoritative. A removal can mean an entry fell out of the top 50,

and does not imply that it was deleted from the inbox. Pagination uses

timestamp and stable event key within a discovery revision boundary.

Read-through boundaries exclude mentions first discovered or reactivated after

that boundary, including older messages resolved by late decryption.

## Executable probes

`tests/mentions_discovery.rs` exercises unopened-room update and cache access,

push-rule behavior for ignored senders and muted rooms, and typed

notification-history pagination.

`tests/mentions_decryption.rs` exchanges a real encrypted event and a delayed

room key between two mock clients, then retrieves the decrypted mention

without opening a UI timeline.

`tests/mentions_checkpoint.rs` holds an event handler pending and verifies

that the SDK has not persisted its position. It cancels the calling task,

then acknowledges the handler and verifies that detached response processing

still saves the position. A new Sliding Sync instance restores that position.

This is the historical event-handler ordering probe. It does not exercise the

production response-commit hook or the durable inbox transaction. It also

demonstrates that returning a handler error still advances the position.

`tests/mentions_inbox.rs` covers live mention classification, read-through

with revision boundaries, independent context limits, and persistence across

session resume. `tests/mentions_late_decryption.rs` verifies that a delayed room key

updates the production inbox once. `src/mentions/service_tests.rs` covers

checkpoint rollback, notification-history recovery, and context updates.

`src/mentions/store/tests.rs` exercises encrypted record restore, capacity,

stale writers, and corruption handling.

These probes establish SDK behavior against mocked responses. They do not

establish homeserver notification retention, room-list completeness, or

complete history on a deployed server.

Focused validation passed 63 native tests and 12 shared inbox and IndexedDB

tests in headless Firefox. Compilation checks also pass without storage

features and for wasm with IndexedDB and mirror support. The full flake gate

was skipped by request.