973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 1 | # macltok request, what a host needs to stream large media itself |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 2 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 3 | ## Problem |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 4 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 5 | `ClientHandle::media(&MediaId, MediaSize)` answers with `Media { bytes: Arc<[u8]>, .. }`, the whole |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 6 | file, decrypted and hash-checked, capped by `EngineConfig::max_media_bytes`. That is right for |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 7 | avatars, thumbnails and images. It is wrong for a 20 minute video. |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 8 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 9 | - In a browser the whole file sits in wasm linear memory, which never shrinks and tops out at 4 GB |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 10 | on wasm32. A few long videos and the worker is carrying hundreds of MB it can never give back. |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 11 | - Nothing plays until the last byte has arrived and the hash has been checked, and seeking has to |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 12 | wait for the same. |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 13 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 14 | malice's on-device build serves media through a service worker so `<video>` and `<audio>` get real |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 15 | URLs and Range requests. For large media it wants to answer each Range itself, fetching the matching |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 16 | ciphertext range from the homeserver's authenticated media endpoint and decrypting it in the service |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 17 | worker with WebCrypto AES-CTR (the counter block is the IV plus `offset / 16`, so any 16-byte aligned |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 18 | range decrypts on its own). Nothing is buffered and the browser caches only ciphertext. The hash can |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 19 | only be checked on a whole read, so a partial read plays unverified bytes. malice accepts that. |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 20 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 21 | What it can't get today is what the request needs. `MediaRegistration` and `MediaSourceInfo` |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 22 | (`normalise/media.rs`) are `pub(crate)`, and there is no way to read the access token. |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 23 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 24 | ## Request |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 25 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 26 | Two read-only accessors on `ClientHandle`. Suggested shapes, adjust freely. |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 27 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 28 | ```rust |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 29 | /// where a registered piece of media lives and how to decrypt it, for a host that fetches it |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 30 | /// itself. `None` for an unknown id, same as `media` answering `MediaUnknown` |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 31 | pub fn media_source(&self, media_id: &MediaId, size: MediaSize) -> Option<MediaSourceDetails>; |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 32 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 33 | #[non_exhaustive] |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 34 | pub struct MediaSourceDetails { |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 35 | pub uri: OwnedMxcUri, |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 36 | pub mimetype: Option<String>, |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 37 | pub declared_size: Option<u64>, |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 38 | /// `Some` for an encrypted attachment |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 39 | pub encryption: Option<AttachmentEncryption>, |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 40 | } |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 41 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 42 | #[non_exhaustive] |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 43 | pub struct AttachmentEncryption { |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 44 | /// the JWK `k`, unpadded url-safe base64, AES-256-CTR |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 45 | pub key: String, |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 46 | /// unpadded base64, the initial counter block |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 47 | pub iv: String, |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 48 | /// unpadded base64 SHA-256 of the ciphertext, for a host that does read it whole |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 49 | pub sha256: String, |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 50 | } |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 51 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 52 | /// the current access token and the homeserver it belongs to. it changes on every refresh, so |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 53 | /// ask again after a 401 rather than keeping it |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 54 | pub fn media_credentials(&self) -> Option<MediaCredentials>; |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 55 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 56 | #[non_exhaustive] |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 57 | pub struct MediaCredentials { |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 58 | pub homeserver: String, |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 59 | pub access_token: String, |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 60 | } |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 61 | ``` |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 62 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 63 | Thumbnail sizes with no uploader thumbnail are server-side resizes, which are never encrypted, so |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 64 | `media_source` for `MediaSize::Thumbnail` can answer the uploader's thumbnail when there is one and |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 65 | the full source otherwise, whichever `media` would have used. |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 66 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 67 | The token accessor is the sensitive one. It hands a caller the bearer token, which `session_record` |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 68 | already does in serialized form, so it doesn't widen what a host can reach. It is worth a doc line |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 69 | saying the token should stay in memory and only go to the homeserver's media endpoints. |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 70 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 71 | ## Also worth considering |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 72 | |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 73 | Streaming decryption inside macltok, `media_stream(&MediaId, range) -> impl Stream<Item = Bytes>`, |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 74 | would let every host stream without reimplementing AES-CTR and Range handling, and would keep the |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 75 | token inside macltok. matrix-sdk 0.18 decrypts attachments whole (`Media::get_media_content`), so |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 76 | it would mean reimplementing the attachment decryptor over the raw download. malice will do it |
973f8fbdaf80 | atagen | 2026-09-28 09:10 UTC | 77 | host-side for now, so this is not blocking. |