cottongin
8ad2f1c800
Includes: - Extended caption styling (font, shadow, dimmed color, bg toggle) - Media server, subtitle downloader, VTT parser, processing modal - Waveform tiers, thumbnail/timeline improvements, transport controls - Hybrid download model, dependency management, clip export enhancements - 21 chat summaries, 2 implementation plans, 2 design specs Co-authored-by: Cursor <cursoragent@cursor.com>
52 lines
4.2 KiB
Markdown
52 lines
4.2 KiB
Markdown
# Fix Fundamental Performance & Audio Regression
|
|
|
|
**Date:** 2026-09-21 14:51
|
|
**Task:** Fix root causes of sluggish UI and broken audio on long videos
|
|
|
|
## Root Causes Identified
|
|
|
|
### 1. Blob URL for Video (CRITICAL)
|
|
The `VideoPlayer` was fetching the ENTIRE preview MP4 file into JavaScript memory via `fetch()` → `blob()` → `URL.createObjectURL()`. For a 54-minute 360p video (~200-500MB), this:
|
|
- Blocks the UI while the entire file is fetched into memory
|
|
- Doubles memory usage temporarily (file buffer + blob)
|
|
- Makes seeks slow because the browser must parse the blob
|
|
- Was the root cause of sluggish playback, delayed seek response, and overall UI lag
|
|
|
|
The blob URL was originally a workaround for WKWebView audio issues with Tauri's asset protocol, but the tradeoff is unacceptable for any video longer than a few minutes.
|
|
|
|
### 2. getImageData/putImageData Overhead
|
|
The "playhead-only redraw" optimization was using `getImageData()` to snapshot the canvas and `putImageData()` to restore it. On a Retina display, this copies 20+ MB of pixel data per frame — worse than just redrawing the canvas from scratch.
|
|
|
|
### 3. Parallel ffmpeg Subprocesses
|
|
`triggerPostDownloadProcessing` fired all three analysis tasks (waveform, keyframes, thumbnails) simultaneously. Each spawns an ffmpeg subprocess that decodes the full file. Three concurrent ffmpeg processes on a 54-minute video saturate the CPU.
|
|
|
|
### 4. Excessive Waveform Peaks
|
|
Hardcoded at 50,000 peaks regardless of duration. For a 54-minute video, this requires decoding the entire audio track at high resolution. Most of these peaks are never visible at the default zoom level.
|
|
|
|
## Changes Made
|
|
|
|
### `src/lib/components/VideoPlayer.svelte`
|
|
- **Removed blob URL entirely** — now uses `convertFileSrc()` directly (Tauri asset protocol). This streams from disk with zero memory overhead, enabling instant seek on any video length.
|
|
- Changed `preload="auto"` to `preload="metadata"` — only loads metadata and first frames, not the entire file.
|
|
- Removed `loadingBlob` state and associated "Preparing video…" UI state.
|
|
- Audio should work with asset protocol for H.264+AAC in MP4 (the preview format). If it doesn't, we'll investigate the specific WKWebView config rather than working around it with blob URLs.
|
|
|
|
### `src/lib/components/Timeline.svelte`
|
|
- **Removed getImageData/putImageData** snapshot mechanism entirely. All redraws go through a single `drawMainCanvas()` call, coalesced by `requestAnimationFrame`.
|
|
- Removed `lastDrawnState`, `lastFullDrawTime`, `pendingPlayheadDraw`, `drawPlayheadOnly()`.
|
|
- Merged the separate "structural" and "playhead" `$effect`s back into one — `requestAnimationFrame` already coalesces multiple calls per frame.
|
|
- Changed `pendingDraw` and `pendingMinimapDraw` from `$state` to plain `let` — they don't need reactivity and were causing unnecessary tracking overhead.
|
|
- Changed `detailFetchTimer` and `detailFetchInFlight` from `$state` to plain `let` — same reason.
|
|
- Reduced waveform detail peak count from `pw * 4` to `pw * 2` (2 peaks per pixel is sufficient).
|
|
|
|
### `src/lib/stores/videoSession.svelte.ts`
|
|
- **Sequential processing**: Changed `triggerPostDownloadProcessing` from parallel fire-and-forget to `async` sequential execution: waveform → keyframes → thumbnails → captions. Only one ffmpeg process runs at a time.
|
|
- **Scaled waveform peaks**: Changed from hardcoded 50,000 to `Math.min(10000, Math.max(2000, duration * 10))`. A 54-minute video gets 10,000 peaks (vs. 50,000 before). A 2-minute video gets 2,000.
|
|
|
|
## Lessons Learned
|
|
|
|
- **Blob URLs are not a scalable workaround** — they work for small files but are catastrophic for anything over a few minutes. Always use the asset protocol (streaming from disk) for video playback.
|
|
- **getImageData/putImageData is expensive on Retina displays** — the data transfer cost (20+ MB per snapshot) exceeds the cost of just redrawing the canvas from primitives.
|
|
- **Sequential ffmpeg is faster than parallel** for analysis tasks on the same file — the file is read from disk for each, and concurrent processes compete for I/O and CPU. Sequential processing also leaves CPU available for the UI thread.
|
|
- **$state variables for non-reactive bookkeeping** (requestAnimationFrame IDs, timers) add unnecessary tracking overhead.
|