gui-video-clipper/chat-summaries/2026-09-21_14-09-fix-multiclip-caption-support-summary.md

Fix Multi-Clip Workflow + Caption Support

Date: 2026-09-21 14:09
Task: Fix multi-clip creation and add time-synced caption support

Task Description

Two main issues addressed:

1. Multi-clip bug: Pressing I/O keys always edited the selected clip instead of creating new clips when the playhead was outside the clip region.
2. Caption support: Full pipeline for downloading, displaying, and exporting time-synced captions (embedded > external, English preferred, auto-generated deprioritized).

Changes Made

1. Fix Multi-Clip Workflow

src/lib/stores/clips.svelte.ts:

• Added isTimeInsideClip(time, clipId) helper with 0.5s tolerance
• markInPoint(time): Now checks if playhead is inside the selected clip's range. If outside, deselects and sets pendingInPoint (starts new clip). If inside, edits the existing clip.
• markOutPoint(time): Same logic — edits selected clip if inside, creates new clip from pending in-point if outside.

src/lib/components/Timeline.svelte:

• Added selectClip(null) call when clicking empty timeline space (no hitTestClip result), so the next I/O presses create a new clip.

2. Caption Metadata Parsing

src-tauri/src/services/video_resolver.rs:

• Extended YtDlpJson struct with subtitles and automatic_captions fields (both HashMap<String, Vec<SubtitleFormat>>)
• Added detect_captions() function: checks for English subtitles (en, en-US, en-GB variants), prioritizes manual over auto-generated
• Added SubtitleFormat struct for deserialization
• Added 4 new unit tests for caption detection

src-tauri/src/models.rs:

• Added has_captions: bool and captions_are_auto: bool to VideoMetadata

src/lib/bindings/video.ts:

• Added hasCaptions and captionsAreAuto to VideoMetadata interface

3. Caption Download

src-tauri/src/services/subtitle_downloader.rs (new file):

• Downloads English VTT subtitles via yt-dlp --write-subs (manual) or --write-auto-subs (auto-generated)
• Uses --sub-langs en.*,en --sub-format vtt --convert-subs vtt
• Searches output directory for .en.vtt files, falls back to any .vtt

src-tauri/src/commands/video.rs:

• Added download_subtitles Tauri command

src/lib/bindings/video.ts:

• Added downloadSubtitles() binding

4. Embedded Subtitle Check

src-tauri/src/commands/media_analysis.rs:

• Added check_embedded_subtitles command that uses ffprobe -show_streams -select_streams s to detect subtitle streams, then extracts as VTT via ffmpeg -map 0:s:0 -f webvtt

src/lib/bindings/mediaAnalysis.ts:

• Added checkEmbeddedSubtitles() binding

5. Caption Display in Player

src/lib/stores/videoSession.svelte.ts:

• Added hasCaptions, captionsAreAuto, captionFilePath to session state
• Added loadCaptions() function that checks embedded subs first, then downloads external
• Integrated into triggerPostDownloadProcessing()

src/lib/components/VideoPlayer.svelte:

• Loads VTT file as blob URL via captionBlobUrl
• Renders <track> element with proper srclang, label (with "(auto)" suffix), and default attribute
• Added CC toggle button (bottom-right overlay) with active/inactive styling
• Syncs textTracks[].mode with captionsEnabled state

6. Caption Export

src-tauri/src/models.rs:

• Added include_captions: bool and caption_file_path: Option<String> to ExportConfig

src-tauri/src/services/clip_exporter.rs:

• Added build_ffmpeg_args_with_subs(): lossless → mux as mov_text/srt; precise → burn-in via subtitles= filter
• Added export_single_clip_with_subs() with fallback to no-subs on failure
• Added export_merged_with_subs()
• Added 2 new unit tests for subtitle-aware arg building

src-tauri/src/commands/export.rs:

• Updated to conditionally use _with_subs variants based on include_captions

src/lib/bindings/export.ts:

• Added includeCaptions and captionFilePath to ExportConfig interface

src/lib/components/ExportDialog.svelte:

• Added "Captions" section with checkbox toggle (only shown when captions are available)
• Shows "(auto-generated)" label when applicable
• Shows mux vs. burn-in note based on cut mode

Infrastructure

src-tauri/src/services/mod.rs: Registered subtitle_downloader module
src-tauri/src/lib.rs: Registered download_subtitles and check_embedded_subtitles commands

Build & Test Results

• Rust: 34 tests pass (including 4 new caption tests + 2 new export tests)
• Frontend: Builds cleanly (0 errors)
• Rust build: Compiles with only pre-existing unused variant warnings

Lessons Learned

• yt-dlp --dump-json includes subtitles and automatic_captions maps — both map language codes to arrays of {ext, url} objects
• WKWebView <track> elements work with blob URLs but need textTracks[].mode managed manually to sync with a toggle
• For ffmpeg subtitle burn-in, paths with colons need escaping (\:) in the subtitles= filter
• Subtitle muxing in lossless mode requires separate -i for the subtitle file and explicit stream mapping (-map 0:v -map 0:a -map 1:s)

Follow-up Items

• The SubtitleFormat.ext and SubtitleFormat.url fields generate "never read" warnings — they exist for serde deserialization but could be suppressed with #[allow(dead_code)]
• Caption time offset accuracy: when using -ss before -i, subtitle timestamps may need adjustment for precise alignment in edge cases
• Sprite sheet optimization (noted from prior session) is still pending