macltok

public

Star0

Fork 0

Watch0

93 commits 3 branches 0 tags 0 releases3.0 MiB

Commit `cff1b0a4eb30`

tarball

atagen <boss@atagen.co>2026-10-25 13:25 UTC

1 file changed+78-0

docs: record the search design

Modified file: docs/design-decisions.md+78 additions@@ -463,6 +463,84 @@ most. a diff for a timeline with no reset is a divergence too, so a wrapper
 that swallowed the reset still heals. anything a late subscriber needs, failures
 included, lives in the state rather than only in an update
 
+## search
+
+`search_messages` takes words and a scope of one room, a list of them or every
+joined room, and answers newest first. it opens a search the way a jump opens an
+event context, a session owning the media of every hit it hands out under its
+own bounded scope, paged with `search_more` and its cursor and freed with
+`close_search`. at most eight stay open and opening another closes the oldest,
+so a host that never closes anything still holds a bounded amount. a hit's
+media ids stop resolving once its search closes. encrypted rooms can't use
+the server's `/search`, so everything goes through one local index, unencrypted
+rooms too, so results and coverage mean the same thing everywhere. a hit comes
+back the way event context shows its target, latest edit applied, with byte
+ranges of the matched words in its text
+
+the index is tantivy, but not the fork's matrix-sdk-search. that crate keeps its
+schema private, keeps an index per room, opens a fresh `IndexWriter` for every
+event, ranks by score, hands back bare event ids and turns mmap on, which wasm
+can't build. `IndexWriter` can't run in wasm either, since it spawns indexing
+threads and rayon pools. so macltok keeps its own segments. each flush writes an
+immutable segment with `SingleSegmentIndexWriter` into a `RamDirectory`, edits
+and redactions are tombstones in a bitset beside it, and segments of a size
+class merge through `merge_filtered_segments` once six pile up, dropping the
+tombstoned docs as they go. all of that is single threaded with the docstore's
+dedicated compressor off. body text is indexed, not stored, a preview reads the
+event back from the event cache or the server. a 2026-10-25 spike ran the
+whole cycle in a headless firefox worker
+
+it lives with the store, per login and never synced. natively it's
+`macltok-search.sqlite3` beside the sdk's databases, in a browser the
+`::macltok-search` IndexedDB database, and both go when the store is deleted.
+every segment, every tombstone generation and the manifest is a blob sealed by
+a `StoreCipher` under the store passphrase, its own key sealed inside so blobs
+can't be swapped. one commit is one transaction, so a crash keeps the last
+commit whole. the working copy lives in memory, all of it, which is the price
+of tantivy's directory being synchronous and IndexedDB not. a store that won't
+open or read means an index in memory for this engine, and coverage says
+`persistent: false`
+
+it's fed from what the client sees. each joined room's event cache is followed
+from its first update, so sync, pagination and the redecryptor's replacements
+all arrive as diffs, and a room idle for five minutes is let go so the cache can
+shrink. an edit lands on its original under the original's key, the latest by
+the original's sender winning, and one that arrives first holds the key
+provisionally until the original turns up. a redaction drops the key and is
+remembered for a while, so a stale copy can't bring it back. redacting the
+latest edit reads the original again. a message still encrypted waits, raw, up
+to two thousand of them, and is decrypted when a room key for its session
+arrives
+
+backfill is `/messages` straight from the room, not through the event cache,
+so it never grows a timeline someone has open. one request at a time across
+every room, two seconds apart, a hundred events a page and ten pages a room per
+engine run, round robin. a room walks back from the newest event to the start,
+and resumes from its saved token next time. the first time a room shows up in
+a run it also gets a head pass from the newest event back until it meets
+something indexed, which picks up what happened while we were offline. failures
+back off from a minute to fifteen. all of that is `EngineConfig`, and zero pages
+a room turns backfill off
+
+coverage is per room, the oldest and newest indexed message, whether backfill
+reached the start, and where backfill is at. it comes with every page
+
+the analyser splits on anything that isn't alphanumeric, lowercases and folds
+accents, so `cafe` finds `Café`. it doesn't stem or segment cjk, a run of cjk is
+one token. words all have to match, quotes make a phrase and `-` excludes.
+tantivy's merger times itself with `Instant::now()` when a `log` logger is on at
+debug, which panics in wasm. so in wasm macltok caps `log` at info at compile
+time, `max_level_info` and `release_max_level_info` both, since in release the
+release cap wins. a host asking for any other cap gets log's own "multiple
+max_level_* features set" error rather than a quiet panic later
+
+a hit is read back when it's shown, `load_or_fetch_event` for the original and
+for its latest edit, which is the event cache and its store first and the
+server's `/event` after. one that's turned out redacted is dropped from the page
+and from the index. one that can't be read at all, gone from the cache with the
+server unreachable or refusing, is left out of that page but stays indexed, so
+it shows up again once it can be read. the cursor still moves past it
+
 ## the sdk fork
 
 we build against a fork of matrix-rust-sdk, branch `macltok`, cut from the

macltok

Commit cff1b0a4eb30

Commit `cff1b0a4eb30`