gui-video-clipper/chat-summaries/2026-09-21_15-10-preprocess-waveform-tiers-summary.md at master

Files

cottongin 8ad2f1c800 chore: stage all pending work — caption styling, media server, processing modal, docs, summaries

Includes:
- Extended caption styling (font, shadow, dimmed color, bg toggle)
- Media server, subtitle downloader, VTT parser, processing modal
- Waveform tiers, thumbnail/timeline improvements, transport controls
- Hybrid download model, dependency management, clip export enhancements
- 21 chat summaries, 2 implementation plans, 2 design specs

Co-authored-by: Cursor <cursoragent@cursor.com>

2026-09-22 10:48:16 -04:00

4.3 KiB

Raw Permalink Blame History

Date: 2026-09-21 15:10
Task: Replace on-demand waveform range extraction with one-time multi-tier pre-processing pipeline, add a processing modal, eliminate the reactive $effect loop bug.

Problem

The $effect in Timeline.svelte that fetched waveform ranges on demand created an infinite reactive loop: writing to session.waveformDetailPeaks re-triggered the same $effect, which re-evaluated needsDetail, which fired again. The terminal showed waveform-range calls repeating 40+ times for the same range, killing performance.

Changes Made

1. Rust Backend — Waveform Tiers (`waveform-tiers-rust`)

src-tauri/src/models.rs: Added WaveformTiers struct with tier0 (~2K peaks), tier1 (~10K peaks), tier2 (all raw peaks, 50K–200K depending on duration).
src-tauri/src/services/waveform_generator.rs: Rewrote entirely. New extract_waveform_tiers(file_path, duration) function runs ffmpeg once at a sample rate yielding ~100 peaks/sec (capped 50K–200K), then downsamples into 3 tiers in Rust. Added downsample() helper (max-of-chunk). Removed extract_waveform_range function. Added 6 unit tests for downsample and compute_peaks.
src-tauri/src/commands/media_analysis.rs: Replaced extract_waveform and extract_waveform_range commands with single extract_waveform_tiers command.
src-tauri/src/lib.rs: Updated handler registration — extract_waveform_tiers replaces extract_waveform + extract_waveform_range.

2. Frontend — Tier-Based Rendering (`waveform-tiers-frontend`)

src/lib/bindings/mediaAnalysis.ts: Added WaveformTiers interface and extractWaveformTiers binding. Removed extractWaveformRange binding.
src/lib/timeline/waveformRenderer.ts: Rewrote. WaveformData is now { tiers: WaveformTiers | null }. pickTier() selects the lowest-resolution tier with ≥1 peak per pixel for the visible range — pure arithmetic, zero IPC.
src/lib/timeline/renderer.ts: Updated drawTimeline signature and waveform condition to use waveform.tiers.
src/lib/stores/videoSession.svelte.ts: Replaced waveformPeaks, waveformDetailPeaks, waveformDetailStart, waveformDetailEnd with waveformTiers: WaveformTiers | null. Added processingStep state. Processing pipeline now calls extractWaveformTiers instead of extractWaveform.
src/lib/components/Timeline.svelte: Deleted the entire $effect for waveform range fetching — the infinite loop bug is eliminated. Removed detailFetchTimer, detailFetchInFlight, needsDetail, extractWaveformRange import. waveformData is now simply { tiers: session.waveformTiers }.

3. Processing Modal (`processing-modal`)

src/lib/components/ProcessingModal.svelte (new): Modal overlay showing processing steps with icons (✓ done, ⏳ current, ○ pending). Steps: downloading → waveform → keyframes → thumbnails → captions → done. Includes download progress bar.
src/App.svelte: Imported and renders <ProcessingModal /> when session.processingStep is neither 'idle' nor 'done'.
src/lib/stores/videoSession.svelte.ts: processingStep is set at each stage of triggerPostDownloadProcessing and beginDownload.

Verification

Rust: cargo build succeeds (3 pre-existing warnings). cargo test passes all 37 tests (including 6 new waveform generator tests).
Frontend: svelte-check reports only pre-existing vite.config.ts errors and a11y warnings — no new issues.

Lessons Learned

The on-demand $effect → async IPC → write reactive state → re-trigger $effect pattern is fundamentally broken in Svelte 5. The correct approach is pre-computation: do all CPU work up front, store the results, and let the renderer pick the right data with pure arithmetic.
For waveform at typical zoom levels (even 100x on an 800px-wide canvas), 50K peaks gives ~6 peaks/pixel, which is plenty. No need for on-demand range extraction.
Downsampling by max-of-chunk preserves peak amplitude fidelity while drastically reducing array size.

Follow-up Items

Audio playback may need verification after the session store changes (the previewFilePath → convertFileSrc pipeline is unchanged, but should be tested).
Consider persisting waveform tiers to disk cache for instant reload on re-open.

4.3 KiB Raw Permalink Blame History Unescape Escape

Pre-Process Waveform Tiers + Processing Modal