macltok

public

Star0

Fork 0

Watch0

95 commits 3 branches 0 tags 0 releases3.0 MiB

Commit `e521a7177da3`

tarball

atagen <boss@atagen.co>2026-10-25 13:25 UTC

1 file changed+73-43

docs: record the search review fixes and the log cap decision

Modified file: docs/design-decisions.md+73 additions−43 deletions@@ -470,9 +470,11 @@ joined room, and answers newest first. it opens a search the way a jump opens an
 event context, a session owning the media of every hit it hands out under its
 own bounded scope, paged with `search_more` and its cursor and freed with
 `close_search`. at most eight stay open and opening another closes the oldest,
-so a host that never closes anything still holds a bounded amount. a hit's
-media ids stop resolving once its search closes. encrypted rooms can't use
-the server's `/search`, so everything goes through one local index, unencrypted
+so a host that never closes anything still holds a bounded amount, and the
+closed one goes out as `SearchClosed`. a hit's media ids carry their search's
+owner prefix like a context's, and stop resolving once it closes. encrypted
+rooms can't use the server's `/search`, so everything goes through one local
+index, unencrypted
 rooms too, so results and coverage mean the same thing everywhere. a hit comes
 back the way event context shows its target, latest edit applied, with byte
 ranges of the matched words in its text
@@ -486,60 +488,88 @@ immutable segment with `SingleSegmentIndexWriter` into a `RamDirectory`, edits
 and redactions are tombstones in a bitset beside it, and segments of a size
 class merge through `merge_filtered_segments` once six pile up, dropping the
 tombstoned docs as they go. all of that is single threaded with the docstore's
-dedicated compressor off. body text is indexed, not stored, a preview reads the
-event back from the event cache or the server. a 2026-10-25 spike ran the
+dedicated compressor off. message text is indexed and not stored, a preview
+reads the event back. the one exception is an edit waiting for its original,
+whose text is kept so the original can take it over. a 2026-10-25 spike ran the
 whole cycle in a headless firefox worker
 
-it lives with the store, per login and never synced. natively it's
-`macltok-search.sqlite3` beside the sdk's databases, in a browser the
-`::macltok-search` IndexedDB database, and both go when the store is deleted.
-every segment, every tombstone generation and the manifest is a blob sealed by
-a `StoreCipher` under the store passphrase, its own key sealed inside so blobs
-can't be swapped. one commit is one transaction, so a crash keeps the last
-commit whole. the working copy lives in memory, all of it, which is the price
-of tantivy's directory being synchronous and IndexedDB not. a store that won't
-open or read means an index in memory for this engine, and coverage says
-`persistent: false`
+it lives with the store, so it's per login and never synced, and it's only
+opened for the user and homeserver that made it. natively it's
+`macltok-search.sqlite3` beside the sdk's databases, created 0600, and in a
+browser the `::macltok-search` IndexedDB database, and both go when the store is
+deleted. every segment, every tombstone generation, the manifest and the waiting
+undecryptable events are blobs sealed by a `StoreCipher` under the store
+passphrase, stored under a keyed hash of their name like the mentions rows, the
+name sealed inside so blobs can't be swapped. the manifest a commit saves is the
+index as it stands once that commit lands, and blobs it doesn't point at are
+dropped on the next load. one commit is one transaction, sqlite in wal with
+normal sync and IndexedDB relaxed, so a crash can lose the last few seconds of
+commits but never leaves half of one. the working copy lives in memory, all of
+it, which is the price of tantivy's directory being synchronous and IndexedDB
+not. a store that won't open or read, or five commits failing in a row, means
+an index in memory for the rest of the engine, and coverage says
+`persistent: false` with the reason in `in_memory`
 
 it's fed from what the client sees. each joined room's event cache is followed
 from its first update, so sync, pagination and the redecryptor's replacements
 all arrive as diffs, and a room idle for five minutes is let go so the cache can
-shrink. an edit lands on its original under the original's key, the latest by
-the original's sender winning, and one that arrives first holds the key
-provisionally until the original turns up. a redaction drops the key and is
-remembered for a while, so a stale copy can't bring it back. redacting the
-latest edit reads the original again. a message still encrypted waits, raw, up
-to two thousand of them, and is decrypted when a room key for its session
-arrives
+shrink. the generic update channel is followed from before the index loads, and
+if it lags every joined room is followed, since a lost update could be a room's
+only one. an edit lands on its original under the original's key, the latest by
+the original's sender winning. one that arrives first waits under its target and
+its own sender, out of results, until the original turns up and takes the
+author's edit, so a stranger's early edit can't crowd it out. an edit of an edit
+is ignored, and an edit to content we can't read clears the old text. a
+redaction drops the key and is remembered for a while, so a stale copy can't
+bring it back. redacting the latest edit reads the original again with whatever
+edits of it are still around. a message still encrypted waits, raw, up to two
+thousand of them in sixteen blobs, and is decrypted eight at a time when a room
+key for its session arrives
 
 backfill is `/messages` straight from the room, not through the event cache,
-so it never grows a timeline someone has open. one request at a time across
-every room, two seconds apart, a hundred events a page and ten pages a room per
+so it never grows a timeline someone has open, filtered to messages, encrypted
+events, redactions, stickers and polls. one request at a time across every
+room, two seconds apart, a hundred events a page and ten pages a room per
 engine run, round robin. a room walks back from the newest event to the start,
-and resumes from its saved token next time. the first time a room shows up in
-a run it also gets a head pass from the newest event back until it meets
-something indexed, which picks up what happened while we were offline. failures
-back off from a minute to fifteen. all of that is `EngineConfig`, and zero pages
-a room turns backfill off
+and resumes from its saved token next time. a room that's been indexed before
+also gets a head pass the first time it shows up in a run, from the newest event
+back until it meets something that was already indexed when the engine started,
+which picks up what happened while we were offline. a failing room backs off on
+its own from a minute to fifteen. a saved token the server turns down with a 400
+is dropped and the room starts again from the newest. leaving a room cancels its
+page. all of that is `EngineConfig`, and zero pages a room turns backfill off
 
 coverage is per room, the oldest and newest indexed message, whether backfill
-reached the start, and where backfill is at. it comes with every page
-
-the analyser splits on anything that isn't alphanumeric, lowercases and folds
-accents, so `cafe` finds `Café`. it doesn't stem or segment cjk, a run of cjk is
-one token. words all have to match, quotes make a phrase and `-` excludes.
-tantivy's merger times itself with `Instant::now()` when a `log` logger is on at
-debug, which panics in wasm. so in wasm macltok caps `log` at info at compile
-time, `max_level_info` and `release_max_level_info` both, since in release the
-release cap wins. a host asking for any other cap gets log's own "multiple
-max_level_* features set" error rather than a quiet panic later
+reached the start, and where backfill is at, plus the undecryptable messages in
+scope. it comes with every page
+
+the analyser makes a token of every run of letters and digits up to 64 chars,
+so hashes and long words fit, and lowercases and folds accents, so `cafe` finds
+`Café`. han, kana, hangul and thai have no spaces to split on, so every char is
+a token and so is every pair of neighbours, and a query for a run needs all its
+pairs. that finds a word inside a sentence at the cost of the odd match across a
+word boundary. emoji are tokens too. queries have no field syntax, so `re:` or a
+url is just its words. words all have to match, quotes make a phrase and `-`
+excludes, and a query with nothing left to find is `Invalid` rather than
+everything in scope. there's no stemming
 
 a hit is read back when it's shown, `load_or_fetch_event` for the original and
 for its latest edit, which is the event cache and its store first and the
-server's `/event` after. one that's turned out redacted is dropped from the page
-and from the index. one that can't be read at all, gone from the cache with the
-server unreachable or refusing, is left out of that page but stays indexed, so
-it shows up again once it can be read. the cursor still moves past it
+server's `/event` after, eight at a time. the original's bundled edit is ignored,
+since the index already knows the newest one. one that's turned out redacted is
+dropped from the page and from the index. one that can't be read at all, gone
+from the cache with the server unreachable or refusing, is left out of that page
+but stays indexed, so it shows up again once it can be read. the cursor still
+moves past it
+
+tantivy's merger times itself with `Instant::now()` whenever a `log` logger is on
+at debug, and that panics in wasm. on 2026-10-25 we capped `log` at info at
+compile time in wasm, `max_level_info` and `release_max_level_info` both, since
+in release the release cap wins. it's all or nothing for the whole wasm build, so
+a host gives up `log` debug and trace from every crate, not just tantivy, and
+asking for any other cap is log's own "multiple max_level_* features set" error.
+tantivy timing with something wasm can call, or reading `Instant` only behind
+its own feature, reopens it
 
 ## the sdk fork

macltok

Commit e521a7177da3

Commit `e521a7177da3`