monolith

public

Star0

Fork 0

Watch0

54 commits 1 branch 0 tags 0 releases1.5 MiB

Commit `5cd954fce34b`

tarball

atagen <boss@atagen.co>2026-10-16 22:46 UTC

4 files changed+83-1

render-vk: absorb the periodic NVIDIA submit stall after flips

NVIDIA stalls for milliseconds on every 1024th vkQueueSubmit2, which dropped a frame every 6.2 s at 165 Hz. Burning the stalling submission on empty command buffers right after a flip moves the stall into idle time. Moving submission to a thread instead would still have delayed whichever frame hit it.

Modified file: crates/monolith/src/drm.rs+6 additions@@ -457,6 +457,12 @@ fn on_vblank(state: &mut State, crtc: crtc::Handle, meta: &mut Option<DrmEventMe
         tracing::warn!("frame was not submitted: {e}");
     }
     surface.queued = false;
+    if let Err(e) = state.renderer.absorb_submit_stall() {
+        tracing::warn!("absorbing a submit stall failed: {e}");
+    }
+    let Some(surface) = state.drm.as_mut().and_then(|d| d.surfaces.get_mut(&crtc)) else {
+        return;
+    };
     if let Some(DrmEventTime::Monotonic(t)) = meta.as_ref().map(|m| m.time) {
         let t = t.as_nanos() as u64;
         if let Some(target) = surface.target_ns.take() {

Modified file: crates/render-vk/Cargo.toml+1

@@ -16,3 +16,4 @@ gpu-allocator = { version = "0.27", default-features = false, features = ["vulka

1616 profiling = "1"

1717 smithay.workspace = true

1818 thiserror = "2"

19+tracing = "0.1"

Modified file: crates/render-vk/src/renderer.rs+75 additions@@ -23,10 +23,12 @@ use smithay::reexports::wayland_server::protocol::wl_buffer::WlBuffer;
 use smithay::utils::{Buffer, Physical, Rectangle, Size, Transform};
 use smithay::wayland::compositor::SurfaceData;
 use smithay::wayland::shm::{shm_format_to_fourcc, with_buffer_contents};
+use std::cell::Cell;
 use std::collections::HashMap;
 use std::fmt;
 use std::marker::PhantomData;
 use std::sync::{Arc, Mutex};
+use std::time::{Duration, Instant};
 
 struct Pending {
     fence: Arc<FenceInner>,
@@ -50,8 +52,43 @@ pub struct VulkanRenderer {
     context: ContextId<VulkanTexture>,
     caps: Arc<DmabufCaps>,
     shared: Arc<Shared>,
+    submissions: Cell<u64>,
+    stall: Cell<StallPhase>,
 }
 
+/// Where in every 1024 submissions the driver stalls. A phase is trusted
+/// once two stalls land on it.
+#[derive(Clone, Copy, Default)]
+struct StallPhase {
+    phase: Option<u64>,
+    seen: Option<u64>,
+}
+
+impl StallPhase {
+    fn saw(self, at: u64) -> Self {
+        if self.seen == Some(at) {
+            if self.phase != Some(at) {
+                tracing::debug!(phase = at, "queue submissions stall periodically");
+            }
+            Self {
+                phase: Some(at),
+                seen: Some(at),
+            }
+        } else {
+            Self {
+                seen: Some(at),
+                ..self
+            }
+        }
+    }
+}
+
+/// A submission this slow is a driver stall rather than work.
+const STALL: Duration = Duration::from_micros(1500);
+/// How many submissions early a stall is absorbed. Each empty one costs
+/// tens of microseconds.
+const ABSORB_AHEAD: u64 = 8;
+
 impl fmt::Debug for VulkanRenderer {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         f.debug_struct("VulkanRenderer")
@@ -111,6 +148,12 @@ impl VulkanRenderer {
             pipelines: Pipelines::new(&shared)?,
             pool,
             spare_cmds: Vec::new(),
+            submissions: Cell::new(0),
+            stall: Cell::new(StallPhase {
+                // Measured on an RTX 3060, see research/nvidia.md.
+                phase: shared.device.name.contains("NVIDIA").then_some(1015),
+                seen: None,
+            }),
             samplers,
             min_filter: TextureFilter::Linear,
             mag_filter: TextureFilter::Linear,
@@ -215,13 +258,45 @@ impl VulkanRenderer {
             .queue
             .lock()
             .unwrap_or_else(|e| e.into_inner());
+        let began = Instant::now();
         {
             profiling::scope!("queue_submit");
             unsafe { d.queue_submit2(*queue, &[submit], fence.fence())? };
         }
+        let n = self.submissions.get();
+        self.submissions.set(n + 1);
+        if began.elapsed() > STALL {
+            self.stall.set(self.stall.get().saw(n % 1024));
+        }
         Ok(fence)
     }
 
+    /// Spends an upcoming stalling submission on empty command buffers, so
+    /// the stall lands now, in idle time, instead of in a frame. Call it
+    /// right after a flip. NVIDIA stalls for milliseconds on every 1024th
+    /// submission, see research/nvidia.md.
+    pub fn absorb_submit_stall(&mut self) -> Result<(), Error> {
+        self.reap();
+        let Some(phase) = self.stall.get().phase else {
+            return Ok(());
+        };
+        let ahead = (phase + 1024 - self.submissions.get() % 1024) % 1024;
+        if ahead >= ABSORB_AHEAD {
+            return Ok(());
+        }
+        for _ in 0..=ahead {
+            let cmd = self.begin()?;
+            let fence = self.end_and_submit(cmd)?;
+            self.pending.push(Pending {
+                fence,
+                cmd,
+                _keep: Vec::new(),
+                _staging: None,
+            });
+        }
+        Ok(())
+    }
+
     /// Records, submits and waits for a short command buffer.
     fn one_shot(
         &mut self,

Modified file: research/nvidia.md+1−1

@@ -10,4 +10,4 @@ Numbers from throwaway benchmarks against monolith's Vulkan renderer, October 20

1010

1111

 `vkQueueSubmit2` normally takes 30 to 90 us, but every 1024th call takes 2.5 to 15 ms, at the same submission indices in every run (1015, 2039, ...). This holds with or without exportable fences and with pooled fences and command buffers, so it is per submission, probably an internal ring wrapping. At 165 Hz it shows up as one dropped frame every 6.2 s.

1212

-Open for M7. Either spend the 1024th submission on an empty submit at a moment with slack, or move submission to its own thread so the render path never waits on it.

+A `vkQueueSubmit2` with no command buffers is a driver no-op that does not advance the count. A submit of an empty command buffer does advance it and takes the stall itself. In a 4200-frame run, absorbing the stall in idle time left zero stalled renders. render-vk now does this after every flip when the stalling submission is within 8, and it learns the phase on other drivers once two stalls land on the same one.

monolith

Commit 5cd954fce34b

Commit `5cd954fce34b`