cottongin
8ad2f1c800
Includes: - Extended caption styling (font, shadow, dimmed color, bg toggle) - Media server, subtitle downloader, VTT parser, processing modal - Waveform tiers, thumbnail/timeline improvements, transport controls - Hybrid download model, dependency management, clip export enhancements - 21 chat summaries, 2 implementation plans, 2 design specs Co-authored-by: Cursor <cursoragent@cursor.com>
48 lines
4.3 KiB
Markdown
48 lines
4.3 KiB
Markdown
# Pre-Process Waveform Tiers + Processing Modal
|
||
|
||
**Date:** 2026-09-21 15:10
|
||
**Task:** Replace on-demand waveform range extraction with one-time multi-tier pre-processing pipeline, add a processing modal, eliminate the reactive `$effect` loop bug.
|
||
|
||
## Problem
|
||
|
||
The `$effect` in `Timeline.svelte` that fetched waveform ranges on demand created an infinite reactive loop: writing to `session.waveformDetailPeaks` re-triggered the same `$effect`, which re-evaluated `needsDetail`, which fired again. The terminal showed `waveform-range` calls repeating 40+ times for the same range, killing performance.
|
||
|
||
## Changes Made
|
||
|
||
### 1. Rust Backend — Waveform Tiers (`waveform-tiers-rust`)
|
||
|
||
- **`src-tauri/src/models.rs`**: Added `WaveformTiers` struct with `tier0` (~2K peaks), `tier1` (~10K peaks), `tier2` (all raw peaks, 50K–200K depending on duration).
|
||
- **`src-tauri/src/services/waveform_generator.rs`**: Rewrote entirely. New `extract_waveform_tiers(file_path, duration)` function runs ffmpeg once at a sample rate yielding ~100 peaks/sec (capped 50K–200K), then downsamples into 3 tiers in Rust. Added `downsample()` helper (max-of-chunk). Removed `extract_waveform_range` function. Added 6 unit tests for downsample and compute_peaks.
|
||
- **`src-tauri/src/commands/media_analysis.rs`**: Replaced `extract_waveform` and `extract_waveform_range` commands with single `extract_waveform_tiers` command.
|
||
- **`src-tauri/src/lib.rs`**: Updated handler registration — `extract_waveform_tiers` replaces `extract_waveform` + `extract_waveform_range`.
|
||
|
||
### 2. Frontend — Tier-Based Rendering (`waveform-tiers-frontend`)
|
||
|
||
- **`src/lib/bindings/mediaAnalysis.ts`**: Added `WaveformTiers` interface and `extractWaveformTiers` binding. Removed `extractWaveformRange` binding.
|
||
- **`src/lib/timeline/waveformRenderer.ts`**: Rewrote. `WaveformData` is now `{ tiers: WaveformTiers | null }`. `pickTier()` selects the lowest-resolution tier with ≥1 peak per pixel for the visible range — pure arithmetic, zero IPC.
|
||
- **`src/lib/timeline/renderer.ts`**: Updated `drawTimeline` signature and waveform condition to use `waveform.tiers`.
|
||
- **`src/lib/stores/videoSession.svelte.ts`**: Replaced `waveformPeaks`, `waveformDetailPeaks`, `waveformDetailStart`, `waveformDetailEnd` with `waveformTiers: WaveformTiers | null`. Added `processingStep` state. Processing pipeline now calls `extractWaveformTiers` instead of `extractWaveform`.
|
||
- **`src/lib/components/Timeline.svelte`**: **Deleted the entire `$effect` for waveform range fetching** — the infinite loop bug is eliminated. Removed `detailFetchTimer`, `detailFetchInFlight`, `needsDetail`, `extractWaveformRange` import. `waveformData` is now simply `{ tiers: session.waveformTiers }`.
|
||
|
||
### 3. Processing Modal (`processing-modal`)
|
||
|
||
- **`src/lib/components/ProcessingModal.svelte`** (new): Modal overlay showing processing steps with icons (✓ done, ⏳ current, ○ pending). Steps: downloading → waveform → keyframes → thumbnails → captions → done. Includes download progress bar.
|
||
- **`src/App.svelte`**: Imported and renders `<ProcessingModal />` when `session.processingStep` is neither `'idle'` nor `'done'`.
|
||
- **`src/lib/stores/videoSession.svelte.ts`**: `processingStep` is set at each stage of `triggerPostDownloadProcessing` and `beginDownload`.
|
||
|
||
## Verification
|
||
|
||
- **Rust**: `cargo build` succeeds (3 pre-existing warnings). `cargo test` passes all 37 tests (including 6 new waveform generator tests).
|
||
- **Frontend**: `svelte-check` reports only pre-existing vite.config.ts errors and a11y warnings — no new issues.
|
||
|
||
## Lessons Learned
|
||
|
||
- The on-demand `$effect` → async IPC → write reactive state → re-trigger `$effect` pattern is fundamentally broken in Svelte 5. The correct approach is pre-computation: do all CPU work up front, store the results, and let the renderer pick the right data with pure arithmetic.
|
||
- For waveform at typical zoom levels (even 100x on an 800px-wide canvas), 50K peaks gives ~6 peaks/pixel, which is plenty. No need for on-demand range extraction.
|
||
- Downsampling by max-of-chunk preserves peak amplitude fidelity while drastically reducing array size.
|
||
|
||
## Follow-up Items
|
||
|
||
- Audio playback may need verification after the session store changes (the `previewFilePath` → `convertFileSrc` pipeline is unchanged, but should be tested).
|
||
- Consider persisting waveform tiers to disk cache for instant reload on re-open.
|