macltok

Persistent mentions discovery

This records the SDK audit and current persistent mentions implementation. Persistent sessions now commit discovery with Sliding Sync responses, retain an encrypted inbox, and expose it through ClientHandle and the mirror. Historical coverage remains incomplete because room-list sync and notification history cannot prove that every past mention was observed.

The audited dependency is Matrix Rust SDK 0.18.0 at fork revision cc2b13ea185c0f694523a3a48d8aadfe366dd767. macltok now pins fork revision 7aace883fbe03d4c3c118b55188fe9519ba8ce48, which adds the response commit hook described below. SDK paths below are relative to the audited revision.

Findings

The SDK can deliver and decrypt events without opening a UI timeline. macltok records observed events and recovers candidates through notification history, but neither source establishes complete historical coverage.

The SDK audit found that an awaited event handler can hold back the Sliding Sync position until a macltok transaction commits, while a background broadcast consumer cannot provide that guarantee. Handlers cannot propagate commit errors, and response processing outlives cancellation of the calling sync task. These limits motivated the fallible response-commit hook now implemented in the pinned SDK fork.

Sync coverage

macltok constructs SyncService in src/engine/mod.rs and uses the SDK room list configuration. The list starts with the selective range 0..=19. After setup it grows by 100 rooms per response, without an SDK-side maximum, until it reaches the server’s room count. Recovery and sufficiently long idle periods can return it to the initial selective range before it grows again. The default list timeline limit is one event per room on initial delivery. Subsequent changes for covered rooms continue arriving through sync.

Opening a room separately subscribes it with a timeline limit of 20. macltok’s update_room_subscriptions and watchers::room_subscriptions manage those explicit subscriptions for active room actors only. The mentions response hook filters entries to joined rooms. The built-in list still includes invited rooms.

Sources are crates/matrix-sdk-ui/src/room_list_service/mod.rs, crates/matrix-sdk-ui/src/room_list_service/state.rs, and crates/matrix-sdk/src/sliding_sync/list/request_generator.rs.

The default one-event initial window cannot recover all earlier pings, including messages sent while the client was offline. Raising SyncServiceBuilder::with_room_list_timeline_limit increases the initial window but does not establish continuous history. The response hook processes joined-room events and records limited timelines of rooms it had already observed, checkpoint discontinuities, and recovery gaps. A room seen for the first time adds no gap, since every room starts that way.

Observation and durable commit

Client::subscribe_to_all_room_updates returns a bounded broadcast receiver of RoomUpdates. Its joined map includes processed timeline events with raw decrypted content when available and optional evaluated push actions. It does not require a UI timeline. Lag is an error that requires recovery, not an invitation to skip to the next update.

Client::observe_events retains only the latest observation and can miss intermediate events. It is unsuitable for durable discovery.

An earlier probe registers Client::add_event_handler for Raw<AnySyncTimelineEvent> with Room context before sync starts. Its awaited handler demonstrates event ordering, but it cannot report commit failures reliably and is not the production ingestion path. Production uses the fallible Sliding Sync response hook described below.

For Sliding Sync v5, the response processor awaits event handlers before updating and persisting pos. The response hook commits the inbox first, so a crash permits replay without skipping committed candidates. Stable event keys make replay idempotent. The hook acknowledgement does not wait for context capture, a UI subscriber, or the engine event stream to drain.

For ordinary event handlers, the SDK logs and discards handler errors, so returning an error after a failed write still permits pos to advance. A pending handler alone also does not provide a reliable shutdown path. The SDK runs response handling in a detached task that outlives cancellation of sync_once or its caller. Aborting the engine’s sync task cannot prevent a suspended handler from later returning and advancing the position. Registering a handler and placing work on a queue without awaiting its commit does not close the crash window.

macltok uses shared pos through its SDK service and commits each response batch through one serialised mentions actor before the hook acknowledges it. The actor records events from joined rooms and from rooms the same response leaves, raw encrypted candidates, limited timeline gaps, and the connection checkpoint in one store commit. Stable room and event keys make replay idempotent. The mentions watch snapshot is replaced before the ack, while mirror and event-stream delivery and context capture run after it. The classic sync next_batch ordering differs and does not establish this guarantee. Nor does this ordering make arbitrary concurrent sync loops safe.

The checkpoint keeps the position the committed response started from as well as the one it ended at. A crash after the commit but before the SDK saves pos replays from that earlier position, and the replay is a superset of what was committed, so it is not treated as a discontinuity. Any other starting position records a SyncDiscontinuity gap.

The hook never fails sync. A failed write keeps the inbox in memory, records a StoreFailed gap, acks, and retries the write a minute later, so a restart before a write lands shows up as a discontinuity. The gap goes again once a later write lands, since nothing was lost. The hook gives up after ten seconds waiting for room in the actor’s queue, drops the response and records HookTimedOut. Once queued, it waits another ten seconds for the commit and then acks regardless, because a queued response still commits in order.

Sources are crates/matrix-sdk/src/event_handler/mod.rs, crates/matrix-sdk/src/event_handler/static_events.rs, crates/matrix-sdk/src/sliding_sync/client.rs, crates/matrix-sdk/src/sliding_sync/mod.rs, and crates/matrix-sdk/src/sliding_sync/cache.rs.

SDK commit hook implementation

The local SDK workspace at ../matrix-rust-sdk-mentions implements the fallible asynchronous response-commit hook through ResponseCommitHook and ResponseCommitBatch. SyncServiceBuilder forwards it only to its room-list connection, so independent notification lookups are unaffected.

The hook receives the processed room events, original room payloads, limited flags, pagination tokens, connection identity, and previous and next position identifiers. It runs before response handlers and observers and before pos is updated or saved. SDK room state may already have been saved. A successful acknowledgement follows the macltok inbox transaction. A failed or dropped acknowledgement returns a sync error without advancing the position.

The hook runs outside SDK store locks and supports local futures on wasm and sendable futures natively. The default absent hook preserves existing SDK behaviour. Detached response processing observes a dropped store acknowledgement after its caller is cancelled. Checkpoint replay works after SDK room state was saved, and stable event keys deduplicate repeated commits.

The SDK audit and focused validation covered commit ordering, failed and dropped acknowledgements, caller cancellation, replay after room-state persistence, restoring a previous checkpoint, absent-hook behaviour, builder forwarding, and connection isolation. The SDK UI library also received a focused wasm compilation check with its js feature. The SDK changes are published to the user’s fork under mentions-commit-hook, and the manifest and both macltok lockfiles pin revision 7aace883fbe03d4c3c118b55188fe9519ba8ce48.

Event cache and delayed decryption

EventCache::subscribe starts global room-update ingestion. The SDK room list service already calls it during construction. Its generic update subscription identifies the affected room, then Room::event_cache and RoomEventCache::find_event can retrieve an event. This does not require a matrix-sdk-ui::Timeline or a retained subscriber per room.

EventCache::subscribe_to_decryption_reports reports resolved event IDs, lag, and backup availability. EventCache::request_decryption accepts room and Megolm session IDs for explicit retries. Room::decrypt_event is also public and can decrypt a retained raw event independently of a UI timeline.

The late-key probe confirms automatic decryption and retrieval of a structured mention through the global cache without a UI timeline. Cached push actions are optional. Re-evaluate them with Room::event_push_actions when needed, and treat missing push context as unresolved rather than a negative result. The room’s own membership state is needed to build that push context.

The cache is not a durable ingestion journal. Its room broadcasts and generic updates can lag, and cache retention can remove events. SQLite and IndexedDB builders select persistent SDK event-cache stores, while the default store-free client uses memory. Even with persistence, the asynchronous cache consumer is not awaited before Sliding Sync saves its position.

The inbox retains unresolved encrypted candidates as encrypted raw events within a configurable bound. It retries due candidates at startup. A report of resolved events retries the encrypted candidates of that room, and lag or a new backup retries all of them. Candidates still unresolved a week after they were first seen expire behind a CandidateExpired gap. Over the bound the oldest events go first, and eviction sets the coverage truncation flag. Imported keys and backups cannot reconstruct events already discarded by every cache.

Sources are crates/matrix-sdk/src/event_cache/mod.rs, crates/matrix-sdk/src/event_cache/redecryptor.rs, crates/matrix-sdk/src/event_cache/caches/room/mod.rs, and crates/matrix-sdk/src/room/mod.rs.

Mention policy

Structured m.mentions.user_ids and m.mentions.room supply semantic candidates. The classifier does not infer pings from rendered bodies or unread counts. A reply relationship alone does not prove that a reply notifies the user, so reply reasons require confirmation that the referenced event belongs to the signed-in user plus either an explicit user mention or a notifying highlight.

Room::event_push_actions evaluates push rules but does not independently filter ignored senders. The service reads the ignore list once per response and again when it changes, and rechecks entries then.

A room mute can suppress evaluated notification actions while structured mention metadata remains present. The classifier retains explicit user mentions in muted rooms and marks notification delivery as suppressed. Room-wide reasons require the relevant push conditions and sender power level. Missing authorisation context remains unresolved for reevaluation. An event that notifies and highlights with no other reason, such as a loud keyword, enters the inbox as Keyword. The SDK’s keyword rules only notify and play a sound, so add_notification_keyword reads the content rule the SDK names after the keyword and adds a highlight tweak when it lacks one, leaving its other actions alone. A rule the user made quiet stays quiet apart from the highlight. A change of reason alone keeps an entry’s saved context. The classifier excludes the signed-in user’s own events even when they contain explicit mention metadata.

The existing NotificationClient combines notification evaluation with ignored-sender filtering. Its public lookup takes known room and event IDs. It is useful for resolving a candidate, but filtered results erase the distinction between mute policy and an absent semantic mention. It is not the sole classifier for this inbox.

Historical candidates and gaps

Ruma exposes api::client::push::get_notifications::v3::Request, sent with Client::send. It accepts opaque from, optional limit, and optional only fields. Responses contain notifications and an optional next_token. macltok requests 50 notifications per page. Entries include raw events, room IDs, timestamps, actions, and server notification read state, which does not initialise or replace local inbox read state.

The service uses this endpoint as a bounded candidate source, validates events after decryption, and applies the live mention policy. Each page’s observed candidates and cursor are committed together in the encrypted inbox, including unresolved encrypted candidates. A failed commit leaves the prior cursor available for replay.

An exhausted notification list does not mean complete room history. Muted-room pings, encrypted mentions not recognised by the server, expired notifications, and server-specific retention can be absent. Recovery inspects at most 20 pages of 50 notifications per run and resumes from committed cursors later. The older-history cursor is separate from the cursor for scans from the newest notifications. A newest scan records an anchor and runs at startup and every 60 seconds. It follows the committed cursor until it reaches that anchor, resuming on later runs when needed to find newly added notifications.

SDK cache gaps retain pagination tokens and support room pagination, but running that pagination mutates the live cache. Mention context capture uses independent /context and /messages requests. Tokens may expire. Context can be retried around a known event while the room remains joined, and known gaps remain visible when continuity cannot be established.

Coverage exposes the earliest observed timestamp, history-endpoint support, recovery activity, known room gaps, unresolved candidate count, and retention truncation. The earliest timestamp is a summary, not proof that every room is covered after that time. Each gap carries a typed MentionGapReason.

Complete offline mention recovery across arbitrary muted and encrypted rooms is not supplied by this endpoint or by the current sync configuration. It would require a retained, replayable event source or bounded per-room history recovery with proof of continuity. The notification endpoint does not provide that guarantee.

Storage, context, and API

src/mentions/store implements encrypted SQLite and IndexedDB record stores. Persistent session construction opens the account store, and local logout deletes it alongside the SDK stores. The mentions actor owns the store for inbox commits and discovery checkpoints.

The passphrase wraps a random SDK StoreCipher. Logical keys are hashed, and authenticated encrypted records include schema, account identity, and logical key. Restore treats mismatched accounts, older schemas, corrupt records, and inconsistent record counts or byte totals as unreadable, empties the store and records a StoreReset gap rather than failing Account::build. A busy or failing store and one written by a newer macltok are left alone, and the inbox runs in memory behind StoreFailed. The busy or failing case is retried every minute, and when it opens, what it held and what was built in memory since are merged. Atomic transactions compare the previous encrypted header before publishing a new revision, so stale writers cannot overwrite a newer checkpoint.

The default budget is 64 MiB of encrypted records, hashed keys, header, and wrapped cipher, excluding database page overhead. The store record cap is 35,128. The inbox defaults to 5,000 entries, 25,000 shared context events, and 5,000 unresolved candidates within that byte budget. When a size or record limit requires eviction, the store drops bounded data and marks coverage truncated.

The storage tests cover atomic restore of records and checkpoints, stale writers, capacity failure, encrypted physical records, resets on wrong credentials or account, corruption and unsupported schemas, and injected SQLite write failure. The inbox tests cover failed writes, checkpoint-only writes, replay from the previous checkpoint, reset of an unreadable inbox, notification recovery, live context updates, and capture persistence across session resume.

The store holds three records. inbox carries entries, saved events, context windows, candidates, edits, redactions, coverage and recovery cursors. sync carries the connection checkpoints and the inbox revision it belongs to. rooms carries the rooms already observed, which drop out when left. Every response writes sync, and inbox or rooms only when the response changed them, all in one commit. Whether the inbox changed is decided by comparing the rooms the response touched, so a quiet response costs those rooms and one small record, while a changed inbox is still reserialised and re-encrypted whole, in one serialisation pass. Background completions are applied in memory and written together at most once a second, and published on that tick even while writes back off. When the budget is short, eviction runs against a size estimate and the inbox is written once. Decrypted previews, saved context, raw candidates, pending capture state, media source registrations, read state, coverage, and sync checkpoints stay inside the authenticated encrypted payloads. SDK SQLite custom values remain plaintext and are not used for decrypted mentions.

Context capture in src/engine/context.rs sizes the before and after requests independently and retains the media registrations produced by normalisation. Each capture requests up to 10 events on each side. At most two capture jobs run concurrently, with up to 128 pending or failed entries considered for scheduling at a time. Capture uses independent /context and /messages requests, so it does not paginate the live cache. Media source registrations are encrypted with saved events and registered again when context is read. Media bodies and key material do not enter the mentions public model.

One account-scoped mentions actor serialises store commits for entries, unresolved candidates, shared context events, read state, revisions, discovery checkpoints, and retention decisions.

Pending capture state is durable with the inbox, so interrupted capture can resume after startup. When a room leaves the joined state, saved context remains readable and its capture state becomes unavailable. Refetch is disabled after departure.

Store publication follows commit. ClientHandle exposes page, context, retry, read, read-through, and recovery methods. The public Command, mention models, mirror state, and updates support serde when the serde feature is enabled, so native and browser consumers can transport them. Browser worker integration is a consumer phase after SDK and macltok revision publication. Malice’s worker still needs mention variants in its explicit Request dispatch and TypeScript methods.

Mirror snapshots contain at most 50 entry previews, coverage, unread count, and a committed revision boundary, with no saved context bodies. The watcher coalesces updates and computes upserts and removals relative to the previous bounded snapshot it observed. A missed update therefore leaves the latest snapshot authoritative. A removal can mean an entry fell out of the top 50, and does not imply that it was deleted from the inbox. Pagination uses timestamp and stable event key within a discovery revision boundary. Read-through boundaries exclude mentions first discovered or reactivated after that boundary, including older messages resolved by late decryption.

Executable probes

tests/mentions_discovery.rs exercises unopened-room update and cache access, push-rule behaviour for ignored senders and muted rooms, and typed notification-history pagination.

tests/mentions_decryption.rs exchanges a real encrypted event and a delayed room key between two mock clients, then retrieves the decrypted mention without opening a UI timeline.

tests/mentions_checkpoint.rs holds an event handler pending and verifies that the SDK has not persisted its position. It cancels the calling task, then acknowledges the handler and verifies that detached response processing still saves the position. A new Sliding Sync instance restores that position. This is the historical event-handler ordering probe. It does not exercise the production response-commit hook or the durable inbox transaction. It also demonstrates that returning a handler error still advances the position.

tests/mentions_inbox.rs covers live mention classification, read-through with revision boundaries, independent context limits, and persistence across session resume. tests/mentions_late_decryption.rs verifies that a delayed room key updates the production inbox once. src/mentions/service_tests.rs covers failed and checkpoint-only writes, checkpoint replay, store reset, notification-history recovery, and context updates. src/mentions/store/tests.rs exercises encrypted record restore, capacity, stale writers, and corruption handling.

These probes establish SDK behaviour against mocked responses. They do not establish homeserver notification retention, room-list completeness, or complete history on a deployed server.

Focused validation passed 63 native tests and 12 shared inbox and IndexedDB tests in headless Firefox. Compilation checks also pass without storage features and for wasm with IndexedDB and mirror support. The full flake gate was skipped by request.