macltok

# macltok design decisions

the contract the working tree actually implements. if the code disagrees with

this, one of them is lying and it should be fixed

## two ways in, one way out

state changes are commands. `ClientHandle::send` queues one and hands back a

`Pending`, `execute` queues and waits, and there's an awaited convenience method

for every command. each command answers exactly once with

`Result<CommandOutput, Error>`. drop the `Pending` if you don't care, the

command still runs

pure reads skip the engine and ask the sdk directly through the handle. rooms,

members, power levels, account data, devices, push notification resolution,

and media bytes all live here

the event stream carries state and only state. timelines, history, typing,

rooms, connection, encryption, verification, upload progress, session. no

command results ride on it. drain it from its own task, it's bounded and the

engine waits for you

one error type for everything, `Error { kind, message, challenge, server }`.

match on `kind`. `Interrupted` means the engine stopped, the room closed, or the

task doing the work died before it could answer. `challenge` turns up with

`InteractiveAuthRequired`

`message` is ours and says what we were doing. `server` is the homeserver's

errcode and message, dug out of the source chain when the error is built, so a

host can show "invalid password" without walking the chain. `AccountError`

carries the same, plus a `kind`. neither folds it into `Display`, since anything

printing the chain would say it twice

## engine

the run loop routes and does nothing slow. every task it spawns is registered

with a context. a command task that panics costs its caller an `Interrupted`

and gets logged. a watcher or a room actor that panics takes the engine down,

emitting `Failed` then `Stopped`. natively, anyway. on wasm a panic traps the

task mid poll and nothing is caught. the rest keeps running in an undefined

state and goes silent, no `Failed` and no `Stopped`, so the host hears of it

through `on_panic`, a free function since a panic hook belongs to the whole

instance rather than to one engine

sync restarts itself with backoff after an error or an unexpected stop, and

`ConnectionChanged` shows `Offline` meanwhile. a run that drops again inside

thirty seconds counts as a failure too, or a server that answers `/versions`

but refuses sync would bounce us between online and offline forever. every return to `Online` re-arms

the send queues, since the sdk parks a room's queue on any send failure and

never unparks it. a recoverable failure while online gets the room another go

after a few seconds

logout and a rejected token both stop the engine. `run` resolves to a

`Termination` naming which of those it was, or a shutdown, or every handle

dropping, so the host never works it out from the events. `Shutdown` answers once

`Stopped` has gone out, every one of them if several were queued

`Engine::run` hands back a boxed future. the real one nests deep enough that

every crate spawning it would trip the compiler's query depth

## rooms

one actor per open room. the engine holds a slot per room, `Active` with a

sender or `Closing` with a fifo of opens and closes that arrived while it wound

down. those replay in order once the actor exits. a room you leave, or get

kicked or banned from, closes itself. it tells the engine first, and anything

already queued to it comes back to be answered or replayed, so an open racing a

leave still gets a fresh actor

however an actor ends, short of the engine stopping, its typing goes out empty

and each of its timelines sends `TimelineClosed`. so does a closed thread. a

timeline that never sent a reset never sends a close either

opening sends `TimelineReset` on the stream before it answers. opening an

already open room sends a fresh reset, so a second frontend gets a snapshot

sends, edits and replies run on the actor in order, so they reach the send

queue in the order they were asked for. redactions, reactions and read

receipts are network calls, so they run beside the actor and a slow one

doesn't queue the room behind it. closing lets them finish rather than lie

threads belong to their room's actor. commands for a thread that's still

opening wait behind the open, up to the actor's queue capacity

pagination runs one request at a time per timeline and emits `HistoryChanged`

around each. the sdk remembers where history starts, we don't, because a gappy

sync can unload it again

the actor's watchers are supervised, and a panic in any of them panics the

actor. its command work isn't, a panicking redaction just answers `Interrupted`

and gets logged

## timelines

`TimelineDiff::try_apply` refuses a diff whose indices don't fit your snapshot

and leaves the snapshot alone. `try_apply_all` does the same for a whole batch,

checking every diff against lengths alone first, since that's all a diff's fit

depends on. snapshots are `imbl::Vector`s so keeping one is a pointer copy.

stop applying, `resync`, and carry on from the next `TimelineReset`. resync

kills the old watcher before the new reset goes out, so no stale diff can land

after it

`TimelineId` tells a room from a thread by its root event

every item is an event, a day divider, the read marker or the start. an event

carries its sender with avatar and name ambiguity, and a `TimelineContent`

saying what it actually is. messages keep their msgtype, membership and profile

changes and the common room state changes get their own variants, and anything

else is `OtherState` or `Other` with its type. nothing is flattened into english

in a body field

thread receipts for a thread that isn't open go straight to the room, so

marking a thread read works the same whether its room is on screen or not

replies carry the replied-to event once it's loaded. the watcher asks the sdk

for any it hasn't got, once per event, since the sdk won't on its own

edit, redact and react take an `EventRef`, so your own message can be edited or

reacted to before the server has echoed it back. redacting an unsent one just

cancels it

a custom emote reaction keys on the image's mxc uri and carries its shortcode

(msc4027), so clients without the pack can still label it. `Reaction` says

which image it is and the shortcode any sender gave, and `toggle_reaction`

takes the shortcode to send

## warming up to send

the sdk does all its key work on the send path, in the room's send queue before

the `/send` itself. members synced, a `/keys/query` for anyone untracked or

dirty, then a `/keys/claim` and the room key sent to every device. over a slow

federation that's a minute of nothing. `prepare_to_send` runs all of it ahead

of time through the fork's `Room::prepare_encryption`, the same function the

send queue calls. the share sits behind the sdk's per-room dedup, so a send

landing mid-warm-up waits for it rather than sharing again, and a send after it

finds nothing left to do but the `/send`. a room whose encryption isn't known

yet gets asked, which the send would have done anyway. one warm-up per room at

a time, a second ask joins it. opening a room or starting to type runs one too,

unless `EngineConfig::prepare_to_send` is off. it's on because malice asked,

not because anyone decided

the session rotates after 100 messages or a week, and sharing isn't a message,

so a warm-up is only wasted on a room left idle about that long

## mutations

membership, moderation, power levels, room creation and account data writes go

through one keyed fifo scheduler. same room, same alias, same dm partner or

same account data type run in order, everything else runs at once.

`create_direct` hands back the existing dm when there is one

notifications and encryption each get one serial worker. the encryption worker

owns the verification controller outright

power level changes build one state event from the local mapping, since the

sdk's own patcher is private

## media

media is a read. `ClientHandle::media` takes a `MediaId`, from a retained

timeline or an avatar, through one bounded

download path with the byte cap the sdk's own download lacks, an hour's total

deadline instead of the sdk client's thirty seconds, and a minute's patience

for a stalled stream. concurrent asks for the same bytes share one flight.

a thumbnail the server won't make falls back to the file itself.

`MediaId` parses from a string, since a web client has to round trip it

the cache holds plaintext, keyed like the sdk keys it, so the sdk and macltok

can read each other's entries. that's how your own encrypted uploads come back.

the sdk keys plain and encrypted sources alike, so a plain event pointing at

someone's encrypted file can leave ciphertext in the cache. an encrypted read

checks the cached bytes against the file's hash and decrypts them if they match

media ids are registered as timelines render and forgotten when their timeline

closes, least recently seen first once the cap is hit

avatars are `MediaId`s too, so a host never handles an mxc uri. they aren't

registered, since they live in the room list and member lists for the whole

session and there can be thousands, and the cap would evict ones still on

screen. a plain avatar carries no keys, so its id is its mxc uri and resolves

without the registry. it never expires, and an mxc uri a host got some other way

parses into one. `set_avatar` still takes an mxc uri

streaming decryption inside macltok was declined. in a browser the service

worker answering range requests is a separate context from the worker running

macltok, and streaming through macltok would carry every chunk through wasm

memory, the very thing the host streams to avoid. `media_source` hands over the

url and keys `media` would have used, picked the same way, and `access_token`

the rest. the byte cap doesn't apply, nothing is buffered

uploads are commands that answer once the send queue has them. the fork puts

upload progress on the client-wide send queue channel too, which upstream keeps

for each room's own, so one watcher there follows every attachment in the queue,

leftovers the sdk resumes from last time included, and ends each with one

`MediaUploadFinished`. sent, cancelled, its room forgotten, or

missed when that channel overflowed at the wrong moment, since then there's no

knowing which. a failure isn't an ending, a wedged upload can still be retried.

uploads don't belong to a room's actor, so closing the room leaves them be

that watcher is the only thing speaking about uploads. anything that ever

showed progress or a failure is followed until it gets exactly one

`MediaUploadFinished`, and nothing about it goes out after. the sdk sends

progress from a detached task, so a tick can turn up behind its own upload's

end, and the watcher remembers the last few hundred ends to drop those. the

engine stopping ends nothing, `Stopped` covers it

## sessions and secrets

`Session::login_password`, `begin_oauth` and `restore` share one client builder

with an encrypted store, refresh tokens and threads. the store is sqlite

natively and indexeddb in a browser. secret-bearing inputs don't derive `Clone`

the homeserver a person typed is checked before the client sees it, and a bad

one is `AccountError::InvalidHomeserver`. a bare server name passes untouched,

since that's what sends the sdk to `.well-known`, and a host that "helpfully"

adds `https://` turns `example.org` into a homeserver url and skips discovery.

a url needs a host, no userinfo, and https. plain http needs

`allow_insecure_loopback` and a loopback host. `SoftLogout::password_login` sets

it, since its homeserver passed once already

a `SessionStore` owns one login's record and its store. login saves the first

record, `restore` loads it, and the engine a `Session` builds saves every token

rotation after that, since a refresh token works once and a host that missed one

save has lost the login. the session changes are subscribed the moment the

client exists, because the sdk can refresh during restore before any engine

does. a save runs off the event stream, so a full stream never holds it back,

and `SessionTokensChanged` only goes out once it has landed, or

`SessionSaveFailed` if it didn't. natively the sdk's save callback runs inside

the refresh, before the new token is used, and `save_blocking` writes there and

then. in a browser there's only the async save, so a tab that dies between a

refresh and its save wakes up holding a spent token. `without_persistence`

hands the job back to the host. `EncryptedSessionStore` seals the record with

the sdk's `StoreCipher` under the store's passphrase and replaces it by rename,

so a crash mid save leaves the old one. it keeps the record out of the sdk's

state store because sqlite writes custom values there in the clear, and a

client can't open its store before it knows the homeserver the record holds.

`discard` deletes the record, then the store, since a record without its store

would restore into empty crypto. the host follows `Termination::retain`.

`Nothing` discards. `Store` calls `soft_log_out`, which keeps the store and

swaps the record for one holding the homeserver, user, device and oauth client

id with no tokens. `restore` hands that back as `Restore::SoftLoggedOut`, so a

restart still knows which device to log in again as, and the login overwrites

it. one store directory per login. a login whose first save fails comes back as

`AccountError::Unsaved` holding the live session, so the host retries the save

rather than spending another device

`Accounts` picks the store, so a host never decides which account a login form

means. it's one owner's accounts under one root and one passphrase, a store and

record each, named by a random id. a password login that names a

soft-logged-out account, by localpart or full user id on the same server, logs

back in as that device into that store. the server is compared by name as well

as by url, since a typed `example.org` is stored as wherever discovery led. a

live account it names is `AlreadyLoggedIn`, anyone else gets a new store. oauth

can't know who it's for until it finishes, so it resumes only the device

`SoftLogout::oauth_login` names. a new store is locked before it's listed, so

`list` can sweep stores with no record, left by a login that died or a discard

that didn't finish, without catching one mid login. a browser lists accounts in

an IndexedDB database of their own, a key each, so two tabs adding at once

don't lose one

the set is per owner because resuming needs only the password, and the store

holds the device's e2ee keys. a set shared across people would hand someone who

knows a password the history a fresh login wouldn't get

one engine per store. before the client opens the store we take the platform's

own exclusive lock on it, `File::try_lock` on `macltok.lock` in the store

directory natively and a Web Lock named after the store in a browser. the

engine and every `ClientHandle` share it, so it goes when the last of them

does, since a lingering handle still has the client and its store open. anyone

else, in this process, another one or another tab, gets `StoreInUse` at once,

so the host decides whether to wait, hand over or say so. the os or the browser

lets go when the holder dies. a SharedWorker serving every tab from one engine

never contends for it. Web Locks only exist in a secure context, so a browser

store needs one too

the sdk's cross process lease can't do this job. in matrix-sdk-common 0.18

`CrossProcessLock::try_lock_once` (`cross_process_lock.rs` 389-448) leases for

500ms and renews every 50ms from a spawned task, which gives up for good on one

failed renewal or on finding the lease lapsed and retaken, and tells no one. a

stall past half a second, ordinary in a tab and certain during the record's

PBKDF2, or one `SQLITE_BUSY`, hands the store to a second engine while the

first still thinks it's alone. with our lock keeping everyone else out, clients

run with `CrossProcessLockConfig::SingleProcess` and the sdk's own sections skip

the lease writes and renewal tasks

a pending OAuth login isn't persisted. `begin_oauth` and `finish` have to run in

the same process, so a web host runs the login in a popup or in a worker that

outlives the redirect. the sdk keeps what `finish_login` needs, the PKCE

verifier among it, in `AuthorizationValidationData` inside a `pub(crate)` map

keyed by `CsrfToken` (matrix-sdk 0.18, `authentication/oauth/mod.rs` around

262-266 and 1744-1756). nothing public reads it or puts it back, and

`finish_login` on a fresh client answers `InvalidState`. doing the code grant

ourselves would copy some 250 lines of the sdk's private oauth behaviour and

drift from it every upgrade. patching the sdk to export and import that data is

small, and since we carry a fork now that's the way to do it. not built yet

## room lists, spaces and profiles

`RoomSummary` has what a room list row needs without a second query. topic,

alias, member counts, heroes for nameless rooms, the inviter, where an upgraded

room went, a latest event preview, the joined spaces listing it, and for a

space the children it lists

parents come from one lazily built `SpaceService` shared by every handle.

children come straight from the space's local state, joined or not.

`space_hierarchy` pages through the server's view for browsing rooms you

haven't joined, and holds its place between pages

`own_profile` and `user_profile` read profiles. name and avatar changes are

commands, and uploading an avatar gets the same byte cap as any upload

## notifications and room data

the notification client is built lazily and shares the engine's sync service.

sdk ui types stay behind the boundary

`rooms` normalises on demand with bounded concurrency and sorts by latest event,

then name, then id. `RoomsChanged` is an invalidation, coalesced so one sync

touching fifty rooms is one event. a left room stays in the list with its

membership until it's forgotten. push rule changes never touch room info, so

`NotificationSettingsChanged` brings a `RoomsChanged` along for the modes in

the summaries

## image packs

msc2545 packs come from the user's account data, from room state, and from the

rooms the user enabled everywhere. the room list asks the server for every

room's `im.ponies.room_emotes`, through the fork's extra required state, so

packs sit in the store like any other state and a change reaches us even across

a gap in the timeline. that's every joined room's packs on the first sync,

tens of kilobytes for a big one, and only changes after. asking only for open

rooms would lose the enabled packs of rooms nobody has open

`image_packs` reads all of it from the store and hands back the lot, global

packs and each room's own, and `ImagePacks::for_room` puts a picker's list in

order. an enabled room the user hasn't joined is skipped, there's no state for

it. one bad image costs that image and a pack with nothing usable is dropped,

since pack contents are whatever someone's client wrote. `ImagePacksChanged`

says a pack moved somewhere, from a state event, an account data change or a

membership change taking a room's packs with it. it's an invalidation, and the

mirror refetches on it like it does rooms, keeping the packs in its state

## verification

`Accept`, `Confirm`, `Reject` and `Cancel`. a request another device sends

shows up as `Incoming` and waits for `Accept`, or `Cancel` to turn it down. one

is only taken up while nothing else is in flight. reject tells the other side

the emoji didn't match. the watchers subscribe before they look at the current

state, since the sdk only reports what changes after you ask. the watcher and

the action paths claim the terminal event, so a racing

cancel can't turn `Done` into `Cancelled`

## mirror

the mirror is a consumer like any other, behind the `mirror` feature, and it

takes any stream of events rather than the `EventStream` itself so a host keeps

the raw events without a second channel. state, its sequence number and the

broadcast send all move under one lock, which is how `subscribe` gets a

snapshot and a receiver with nothing between them. a lagged subscriber starts

over from a fresh state under the same lock

nothing on the draining path waits for an engine answer. a resync runs on its

own task, retrying with backoff until a reset or a close for its timeline

aborts it. rooms are fetched on a task too, one at a time with one more owed at

most. a diff for a timeline with no reset is a divergence too, so a wrapper

that swallowed the reset still heals. anything a late subscriber needs, failures

included, lives in the state rather than only in an update

## the sdk fork

we build against a fork of matrix-rust-sdk, branch `macltok`, cut from the

matrix-sdk 0.18.0 tag. turned down on 2026-09-26, taken up on 2026-10-22 once

it was holding back features every peer client has. every workspace that builds

macltok needs the same `[patch.crates-io]` block, since patches don't reach

dependents, and it keeps macltok off crates.io until upstream takes the patches.

a vendored copy of macltok should drop its own block, cargo ignores it there

and says so on every build

it carries visibility and config changes, each small enough to be an upstream

pr, and nothing that changes how the sdk behaves. if that stops being true,

especially in the timeline or the send queue, it's time to drop matrix-sdk and

matrix-sdk-ui for our own layer over matrix-sdk-base and matrix-sdk-crypto,

which keep their public sans-io apis

- `Room::prepare_encryption`, the send path's encryption prep made public

- `SyncServiceBuilder::with_room_list_extra_required_state`, for state the

  sdk's hard-coded list leaves out. image packs use it

- msc4027 reaction shortcodes, kept on the timeline's `ReactionInfo` and sent

  with `Timeline::toggle_reaction_with_shortcode`

- `SendQueue::enable_global_upload_progress`, upload progress on the

  client-wide send queue channel

## public model

owned ruma ids and typed data wherever there's a stable shape, `Raw` where

arbitrary content has to survive. ruma is re-exported as `macltok::ruma`, so a

host parses ids at the version we're typed against. anything expected to grow is

`#[non_exhaustive]`. `Engine::from_client` stays public for people who built

their own client