From 8d669a1a8a7e69db879ccbd27be372e50e2926a2 Mon Sep 17 00:00:00 2001
From: cottongin <cottongin@users.noreply.github.com>
Date: Mon, 21 Sep 2026 10:07:58 -0400
Subject: [PATCH] Update spec: add waveform, thumbnail strip, cookie/auth,
 dependency chain to MVP

Co-authored-by: Cursor <cursoragent@cursor.com>
---
 .../specs/2026-09-21-video-clipper-design.md  | 144 ++++++++++++++++--
 1 file changed, 129 insertions(+), 15 deletions(-)

diff --git a/docs/superpowers/specs/2026-09-21-video-clipper-design.md b/docs/superpowers/specs/2026-09-21-video-clipper-design.md
index 7425cb4..423fa07 100644
--- a/docs/superpowers/specs/2026-09-21-video-clipper-design.md
+++ b/docs/superpowers/specs/2026-09-21-video-clipper-design.md
@@ -31,12 +31,14 @@ Manages all subprocess orchestration, file I/O, and state persistence. Communica
 
 | Component | Responsibility |
 |---|---|
-| `DependencyManager` | Check for ffmpeg/yt-dlp on `$PATH`. If missing, offer to install via `brew install`. Shows a modal on startup if dependencies are absent. |
+| `DependencyManager` | Check the full dependency chain on `$PATH`: ffmpeg/ffprobe, yt-dlp, a JavaScript runtime (Deno preferred, Node.js fallback), and the `bgutil-ytdlp-pot-provider` yt-dlp plugin. On first launch or when anything is missing, shows a step-by-step **Setup Wizard** that walks the user through installing each dependency one at a time. See the Dependency Chain section. |
 | `VideoResolver` | Takes a URL, calls `yt-dlp --dump-json <url>` for metadata and `yt-dlp -g <url>` for the direct stream URL. Returns a `VideoMetadata` struct. |
 | `DownloadManager` | Runs `yt-dlp -o <temp_path> <url>` in the background. Emits progress events to the frontend. Manages temp file lifecycle (cleanup on cancel/error). |
 | `KeyframeIndex` | After download completes, runs `ffprobe -select_streams v -show_entries frame=pts_time,flags -of csv <file>` to extract keyframe positions. Returns a sorted list of keyframe timestamps. |
 | `ClipExporter` | Runs ffmpeg for clip extraction. Supports two modes: lossless (stream copy, keyframe-aligned) and precise (re-encode, exact frames). Handles both individual and merged export. |
-| `PreferencesStore` | Persists user preferences (output directory, window geometry) via `tauri-plugin-store`. |
+| `WaveformGenerator` | After download completes, extracts audio peak data from the video file via ffmpeg. Computes ~8000 peak samples for the full video duration and sends them to the frontend for Canvas rendering. |
+| `ThumbnailExtractor` | After download completes, extracts frames at computed intervals via ffmpeg (1/sec for <10 min, 1/2s for 10–60 min, 1/5s for >60 min). Outputs small JPEGs (160×90px) packed into sprite sheets for efficient rendering. |
+| `PreferencesStore` | Persists user preferences (output directory, window geometry, cookie source) via `tauri-plugin-store`. |
 
 ### 2. Svelte Frontend
 
@@ -48,18 +50,20 @@ Renders the video player, custom timeline, clip management panel, and export dia
 |---|---|
 | `UrlInput` | Text field for URL entry. Detects paste, shows loading spinner during resolution, displays errors. |
 | `VideoPlayer` | Wraps the HTML5 `<video>` element. Exposes play/pause/seek API. Stays synced with the Timeline component via a shared Svelte store. |
-| `Timeline` | Canvas-based timeline. Renders: playhead (current position), clip regions (colored rectangles with draggable start/end handles), keyframe markers (after index is built). Supports click-to-seek, drag-to-scrub, zoom via scroll wheel or slider. |
+| `Timeline` | Canvas-based timeline with four visual layers (top to bottom): thumbnail strip, waveform, clip markers, controls. Renders: playhead (current position), clip regions (colored rectangles with draggable start/end handles), keyframe markers (after index is built), thumbnail frames (after extraction), audio waveform (after extraction). Supports click-to-seek, drag-to-scrub, zoom via scroll wheel or slider. |
 | `TransportControls` | Play/pause, frame step (±1 frame), keyframe jump, seek ±1s/±5s buttons. Wired to keyboard shortcuts. |
 | `ClipList` | Scrollable panel listing all clips. Each row shows: color swatch, label (editable), start time (editable), end time (editable), delete button. Clicking a row selects the clip and jumps the playhead to its start. |
 | `ExportDialog` | Modal dialog for export configuration. See Export Flow section. |
 | `StatusBar` | Bottom bar showing: download progress (percentage + ETA), active export progress, tool availability status. |
-| `DependencyPrompt` | Modal shown on startup if ffmpeg/yt-dlp are missing. Offers to install via Homebrew. |
+| `SetupWizard` | Step-by-step modal shown on first launch or when dependencies are missing. Walks the user through installing each dependency one at a time with green checkmarks for what's already present. See the Dependency Chain section. |
 
 ### 3. External CLI Tools
 
-- **yt-dlp:** URL resolution, metadata extraction, stream URL retrieval, video downloading.
-- **ffmpeg:** Clip extraction (both lossless and re-encode modes), clip merging.
+- **yt-dlp:** URL resolution, metadata extraction, stream URL retrieval, video downloading. All yt-dlp invocations include the user's configured cookie source (see Cookie/Auth section).
+- **ffmpeg:** Clip extraction (both lossless and re-encode modes), clip merging, waveform peak extraction, thumbnail frame extraction.
 - **ffprobe:** Keyframe position extraction (part of the ffmpeg package).
+- **deno** (or **node**): JavaScript runtime required by the PO Token provider plugin for YouTube playback.
+- **bgutil-ytdlp-pot-provider:** yt-dlp plugin that automatically generates per-video Proof-of-Origin tokens required by YouTube.
 
 ## Data Model
 
@@ -82,12 +86,30 @@ interface VideoSession {
   localFilePath: string | null; // set once background download completes
   downloadProgress: number;   // 0.0 - 1.0
   keyframePositions: number[]; // populated after download + ffprobe
+  waveformPeaks: number[];    // normalized audio peaks (0.0–1.0), ~8000 samples, populated after download + ffmpeg
+  thumbnailSpritesheets: ThumbnailSpritesheet[]; // populated after download + ffmpeg
   clips: Clip[];
 }
 
+interface ThumbnailSpritesheet {
+  filePath: string;            // path to the spritesheet image
+  startIndex: number;          // index of first thumbnail in this sheet
+  count: number;               // number of thumbnails in this sheet
+  thumbWidth: number;          // pixel width of each thumbnail (e.g. 160)
+  thumbHeight: number;         // pixel height of each thumbnail (e.g. 90)
+  columns: number;             // thumbnails per row in the spritesheet
+  intervalSeconds: number;     // time interval between thumbnails
+}
+
+type CookieSource =
+  | { type: "browser"; browser: "firefox" }
+  | { type: "file"; path: string }
+  | { type: "none" };
+
 interface UserPreferences {
   outputDirectory: string;    // default: home directory (~/)
   windowBounds: { x: number; y: number; width: number; height: number } | null;
+  cookieSource: CookieSource; // default: { type: "browser", browser: "firefox" }
 }
 ```
 
@@ -109,11 +131,11 @@ interface UserPreferences {
 ├─────────────────────────────────────────────────────────┤
 │  ▼ playhead                                             │
 │  ┃                                                      │
-│  ╠══════════════════════════════════════════════════════╣
-│  ║     ┃  ▓▓▓▓▓▓▓▓▓▓▓  ┃      ┃  ▓▓▓▓▓  ┃           ║
-│  ╠══════════════════════════════════════════════════════╣
+│  ┃ [thumb] [thumb] [thumb] [thumb] [thumb] [thumb]      │ ← Thumbnail strip
+│  ┃ ∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿ │ ← Waveform
+│  ┃     ▓▓▓▓▓▓▓clip 1▓▓▓▓▓▓▓▓▓     ▓▓▓clip 2▓▓▓       │ ← Clip markers
 │  ┃                                                      │
-│  [🔍−] ═══════════●══════════ [🔍+]     zoom slider    │
+│  [🔍−] ═══════════●══════════ [🔍+]     zoom slider    │ ← Controls
 ├─────────────────────────────────────────────────────────┤
 │  CLIPS                                                  │
 │  ┌──────────────────────────────────────────┐           │
@@ -125,6 +147,14 @@ interface UserPreferences {
 └─────────────────────────────────────────────────────────┘
 ```
 
+The timeline area has four visual layers (top to bottom):
+1. **Thumbnail strip** — Video frame thumbnails at adaptive density based on zoom level.
+2. **Waveform** — Audio amplitude visualization.
+3. **Clip markers** — Colored regions with draggable handles, overlaying the above.
+4. **Controls** — Zoom slider and playhead position.
+
+Before the background download completes, the thumbnail and waveform layers show a static placeholder (subtle gradient bar with "Generating…" label). They fade in with real data once extraction finishes.
+
 ### Timeline Interactions
 
 - **Scrub:** Click anywhere on the timeline to jump. Click-and-drag to scrub continuously.
@@ -206,15 +236,100 @@ Export requires the background download to be complete (ffmpeg operates on the l
 - User can wait or cancel.
 - Export begins automatically once the download finishes.
 
+## Dependency Chain & Setup Wizard
+
+YouTube's 2026 anti-bot measures require more than just ffmpeg and yt-dlp. The full dependency chain:
+
+| Dependency | Purpose | Install Method | Check |
+|---|---|---|---|
+| `ffmpeg` (+ `ffprobe`) | Clip extraction, waveform generation, thumbnail extraction, keyframe indexing | `brew install ffmpeg` | `which ffmpeg` |
+| `yt-dlp` | URL resolution, metadata, streaming, downloading | `brew install yt-dlp` | `which yt-dlp` |
+| `deno` (preferred) or `node` | JavaScript runtime for PO Token generation | `brew install deno` | `which deno \|\| which node` |
+| `bgutil-ytdlp-pot-provider` | yt-dlp plugin: automatic per-video Proof-of-Origin token generation for YouTube | `python3 -m pip install -U bgutil-ytdlp-pot-provider` | Run `yt-dlp -v --simulate "https://www.youtube.com/watch?v=dQw4w9WgXcQ"` and check for `GetPOT` in `[debug] Extractor Plugins:` output |
+
+### Setup Wizard UX
+
+On first launch (or when any dependency is missing), the app shows a step-by-step Setup Wizard modal:
+
+1. Each dependency is listed as a row with a status indicator: ✅ (installed), ❌ (missing), or ⏳ (installing).
+2. The wizard checks each dependency in order. For the first missing item, it shows a description of what it does and an **[Install]** button.
+3. Clicking **[Install]** runs the appropriate install command. Output is shown in a scrollable log area within the wizard.
+4. After successful install, the status updates to ✅ and the wizard advances to the next item.
+5. A **[Skip]** button is available for each step (the app will work for non-YouTube sources without the PO token chain, and without cookies for public content).
+6. Once all critical dependencies are present (at minimum: ffmpeg + yt-dlp), the wizard closes.
+
+The wizard also runs a quick self-check on subsequent launches. If everything is present, it's invisible. If something was uninstalled or broken, it resurfaces.
+
+## Cookie / Authentication
+
+YouTube requires session cookies for private, members-only, and age-restricted content. Even for public content, cookies can help avoid rate limiting and bot detection.
+
+### Cookie Source Configuration
+
+In the Preferences panel, the user configures their cookie source:
+
+- **Default: Firefox** (`--cookies-from-browser firefox`). Firefox is the only browser where `--cookies-from-browser` reliably works — Chromium browsers (Chrome 127+) encrypt cookies with a process-bound key that yt-dlp can't decrypt.
+- **Fallback: cookies.txt file** — User points to a Netscape-format cookies.txt file (exported via a browser extension). Useful if Firefox isn't available.
+- **None** — No cookies. Works for public, non-restricted content on non-YouTube sites.
+
+If the user selects Firefox but Firefox isn't installed, the app shows a note: "Firefox is recommended for reliable YouTube authentication. Alternatively, import a cookies.txt file."
+
+### Integration with yt-dlp
+
+Every yt-dlp invocation (metadata, stream URL, download) includes the configured cookie flag:
+- Firefox: `--cookies-from-browser firefox`
+- cookies.txt: `--cookies /path/to/cookies.txt`
+- None: no cookie flag
+
+## Waveform Visualization
+
+### Generation
+
+After the background download completes, the `WaveformGenerator` backend component extracts audio peak data:
+
+1. Run ffmpeg to decode the audio track and compute amplitude peaks at a fixed sample count (~8000 samples for the full video duration).
+2. Normalize peaks to a 0.0–1.0 range.
+3. Send the peak array to the frontend via a Tauri event.
+
+The exact ffmpeg pipeline: decode audio → resample to a low rate matching the desired sample count → extract absolute amplitude values → normalize.
+
+### Display
+
+- **Before download completes:** Static placeholder — a subtle striped gradient bar with a centered "Generating waveform…" label. Clearly communicates that data is coming.
+- **After extraction:** Replace the placeholder with the real waveform, rendered on the Canvas as a filled amplitude graph. Smooth fade-in transition.
+- The waveform responds to zoom: at higher zoom levels, more detail is visible. At lower zoom levels, peaks are aggregated (max value per visible pixel).
+
+## Thumbnail Strip
+
+### Generation
+
+After the background download completes, the `ThumbnailExtractor` backend component extracts video frames:
+
+1. Compute the extraction interval based on video duration:
+   - Short videos (<10 min): 1 frame per second
+   - Medium videos (10–60 min): 1 frame per 2 seconds
+   - Long videos (>60 min): 1 frame per 5 seconds
+2. Run ffmpeg to extract frames at the computed interval, scaled to 160×90px (16:9).
+3. Pack extracted frames into sprite sheets (e.g. 10×10 grids = 100 thumbnails per sheet) for efficient loading and rendering.
+4. Send sprite sheet metadata (file paths, dimensions, interval) to the frontend via a Tauri event.
+
+### Display
+
+- **Before download completes:** Static placeholder — a subtle gradient bar with "Generating thumbnails…" label. Matches the waveform placeholder style.
+- **After extraction:** Replace the placeholder with real thumbnails, drawn from sprite sheets onto the Canvas.
+- **Adaptive density:** The timeline shows ~1 thumbnail per 100px of visible width. Zooming in reveals more granular frames (showing frames closer together in time); zooming out shows fewer (skipping intermediate frames). The extraction provides enough frames to support all reasonable zoom levels; the Canvas renderer picks which frames to display based on the current visible time range and pixel width.
+
 ## Edge Cases
 
 | Scenario | Behavior |
 |---|---|
 | Stream URL expires during preview | Detect playback error on `<video>`, re-resolve via yt-dlp, resume at same position. |
-| Age-restricted / auth-required video | Show error: "This video requires authentication." Guidance points to future cookie support. |
+| Age-restricted / auth-required video | If no cookie source is configured, show error: "This video requires authentication. Configure cookies in Preferences." If cookies are configured but still fail, show the yt-dlp error with guidance. |
 | Overlapping clips | Allowed. Each clip is independent. In merged export, clips are sorted by start time. If clip B starts before clip A ends, clip B's start is trimmed to clip A's end to avoid duplicate frames. |
 | Very long video (>2 hours) | No special limits. Timeline zoom handles precision. |
-| ffmpeg/yt-dlp missing on startup | Modal: "Required tools not found. Install via Homebrew?" → [Install] runs `brew install ffmpeg yt-dlp` with visible output. [Manual] dismisses. |
+| Any dependency missing on startup | Setup Wizard appears, walking user through installing each missing dependency step by step. See Dependency Chain section. |
+| PO Token provider not working | yt-dlp may still work for some content without the PO token, but YouTube will likely block playback. Show a warning: "YouTube access may be limited. Check the PO Token provider in Preferences." |
+| Waveform/thumbnail generation fails | Non-fatal. Log the ffmpeg error, keep the placeholder visible with an unobtrusive "Failed to generate" message. All other features continue working. |
 | Export fails mid-way | Show ffmpeg error output in a dismissable panel. Partial files cleaned up. |
 | Non-YouTube URL | yt-dlp supports 1000+ sites. App tries any URL. Shows "Unsupported URL" if yt-dlp fails. |
 | Network disconnection during download | Download pauses. App shows "Network unavailable" status. Resumes automatically when connection returns (yt-dlp handles resume via `-c`). |
@@ -227,9 +342,8 @@ These are explicitly out of scope for the initial version but should be kept in
 - **Local file support** — Drag & drop a local `.mp4`/`.mkv`/`.webm` file. Skip the yt-dlp step, go straight to playback + timeline.
 - **Video quality/resolution selection** — After URL resolution, show available formats and let the user pick (720p, 1080p, 4K, etc.) before downloading.
 - **Save/load clip projects** — Persist clip data (URL, timestamps, labels) to a project file. Reopen later to continue editing or re-export.
-- **Waveform/audio visualization** — Show audio waveform on the timeline for easier cut-point identification.
-- **Thumbnail strip** — Show video frame thumbnails along the timeline for visual scrubbing.
 - **Batch processing** — Queue multiple URLs, each with their own clips.
-- **Cookie/auth support** — Allow yt-dlp `--cookies-from-browser` or manual cookie file for age-restricted/private videos.
 - **Custom ffmpeg flags** — Power-user option to pass arbitrary ffmpeg arguments during export.
 - **Cross-platform** — Tauri supports Windows/Linux. The architecture doesn't preclude this, but macOS is the target for MVP.
+- **Additional cookie browsers** — Support `--cookies-from-browser` for browsers beyond Firefox (if Chromium cookie encryption issues are resolved upstream).
+- **Progressive waveform** — Build waveform data in real-time via Web Audio API as the stream plays, rather than waiting for download.