gui-video-clipper/chat-summaries/2026-09-21_14-51-fix-fundamental-perf-audio-summary.md

Fix Fundamental Performance & Audio Regression

Date: 2026-09-21 14:51
Task: Fix root causes of sluggish UI and broken audio on long videos

Root Causes Identified

1. Blob URL for Video (CRITICAL)

The VideoPlayer was fetching the ENTIRE preview MP4 file into JavaScript memory via fetch() → blob() → URL.createObjectURL(). For a 54-minute 360p video (~200-500MB), this:

• Blocks the UI while the entire file is fetched into memory
• Doubles memory usage temporarily (file buffer + blob)
• Makes seeks slow because the browser must parse the blob
• Was the root cause of sluggish playback, delayed seek response, and overall UI lag

The blob URL was originally a workaround for WKWebView audio issues with Tauri's asset protocol, but the tradeoff is unacceptable for any video longer than a few minutes.

2. getImageData/putImageData Overhead

The "playhead-only redraw" optimization was using getImageData() to snapshot the canvas and putImageData() to restore it. On a Retina display, this copies 20+ MB of pixel data per frame — worse than just redrawing the canvas from scratch.

3. Parallel ffmpeg Subprocesses

triggerPostDownloadProcessing fired all three analysis tasks (waveform, keyframes, thumbnails) simultaneously. Each spawns an ffmpeg subprocess that decodes the full file. Three concurrent ffmpeg processes on a 54-minute video saturate the CPU.

4. Excessive Waveform Peaks

Hardcoded at 50,000 peaks regardless of duration. For a 54-minute video, this requires decoding the entire audio track at high resolution. Most of these peaks are never visible at the default zoom level.

Changes Made

src/lib/components/VideoPlayer.svelte

• Removed blob URL entirely — now uses convertFileSrc() directly (Tauri asset protocol). This streams from disk with zero memory overhead, enabling instant seek on any video length.
• Changed preload="auto" to preload="metadata" — only loads metadata and first frames, not the entire file.
• Removed loadingBlob state and associated "Preparing video…" UI state.
• Audio should work with asset protocol for H.264+AAC in MP4 (the preview format). If it doesn't, we'll investigate the specific WKWebView config rather than working around it with blob URLs.

src/lib/components/Timeline.svelte

• Removed getImageData/putImageData snapshot mechanism entirely. All redraws go through a single drawMainCanvas() call, coalesced by requestAnimationFrame.
• Removed lastDrawnState, lastFullDrawTime, pendingPlayheadDraw, drawPlayheadOnly().
• Merged the separate "structural" and "playhead" $effects back into one — requestAnimationFrame already coalesces multiple calls per frame.
• Changed pendingDraw and pendingMinimapDraw from $state to plain let — they don't need reactivity and were causing unnecessary tracking overhead.
• Changed detailFetchTimer and detailFetchInFlight from $state to plain let — same reason.
• Reduced waveform detail peak count from pw * 4 to pw * 2 (2 peaks per pixel is sufficient).

src/lib/stores/videoSession.svelte.ts

• Sequential processing: Changed triggerPostDownloadProcessing from parallel fire-and-forget to async sequential execution: waveform → keyframes → thumbnails → captions. Only one ffmpeg process runs at a time.
• Scaled waveform peaks: Changed from hardcoded 50,000 to Math.min(10000, Math.max(2000, duration * 10)). A 54-minute video gets 10,000 peaks (vs. 50,000 before). A 2-minute video gets 2,000.

Lessons Learned

• Blob URLs are not a scalable workaround — they work for small files but are catastrophic for anything over a few minutes. Always use the asset protocol (streaming from disk) for video playback.
• getImageData/putImageData is expensive on Retina displays — the data transfer cost (20+ MB per snapshot) exceeds the cost of just redrawing the canvas from primitives.
• Sequential ffmpeg is faster than parallel for analysis tasks on the same file — the file is read from disk for each, and concurrent processes compete for I/O and CPU. Sequential processing also leaves CPU available for the UI thread.
• $state variables for non-reactive bookkeeping (requestAnimationFrame IDs, timers) add unnecessary tracking overhead.