gui-video-clipper/chat-summaries/2026-09-21_13-45-fix-audio-dynamic-waveform-summary.md

Fix Audio Playback + Dynamic Waveform Resolution

Task Description

1. Fix missing audio on preview playback — Video plays but with no sound in WKWebView.
2. Dynamic waveform resolution — Waveform should increase in detail as the user zooms in.

Root Cause Analysis (Audio)

The preview file is correctly muxed (H.264 640x360 + AAC 128kbps, moov atom at offset 24 — fast-start). WKWebView config analysis:

• wry 0.55.1 defaults autoplay: true → sets mediaTypesRequiringUserActionForPlayback = None
• tauri-runtime-wry uses WebViewBuilder::new_with_web_context() which inherits this default
• Tauri v2.11.6 doesn't expose or override autoplay
• So WKWebView SHOULD be configured for audio playback

Despite correct configuration, WKWebView's Tauri asset protocol (https://asset.localhost/...) appears to silently drop audio tracks when streaming local files — possibly due to range request handling or MIME type issues in the custom protocol handler.

Fix: Load the video file as a blob: URL instead of using the asset protocol. This bypasses the asset protocol entirely and uses WKWebView's native blob URL handling, which reliably plays both audio and video tracks.

Changes Made

Audio Fix: Blob URL Video Loading (src/lib/components/VideoPlayer.svelte)

• Added a $effect that fetches the preview file via convertFileSrc URL, converts it to a Blob, then creates a blob: URL
• Video element now uses the blob URL instead of the asset protocol URL
• Added playsinline attribute to the video element
• Added explicit volume = 1 and muted = false on loadeddata
• Added .play() Promise error handling (catches and logs rejections)
• Shows "Preparing video…" state while blob is loading
• Falls back to asset protocol URL if blob creation fails
• Properly revokes old blob URLs on session change

Dynamic Waveform: Backend (src-tauri/src/services/waveform_generator.rs)

• Added extract_waveform_range(file_path, start_time, end_time, peak_count) function
• Uses ffmpeg -ss {start} -t {duration} with 44100 Hz sample rate for high-resolution extraction
• Computes peaks for just the specified time range

Dynamic Waveform: Command (src-tauri/src/commands/media_analysis.rs)

• Added extract_waveform_range Tauri command
• Registered in lib.rs

Dynamic Waveform: Frontend Binding (src/lib/bindings/mediaAnalysis.ts)

• Added extractWaveformRange(filePath, startTime, endTime, peakCount) binding

Dynamic Waveform: Session State (src/lib/stores/videoSession.svelte.ts)

• Added waveformDetailPeaks, waveformDetailStart, waveformDetailEnd to session state
• Increased initial waveform extraction from 8,000 to 50,000 peaks (good for most zoom levels)
• Detail fields cleared on session reset

Dynamic Waveform: Renderer (src/lib/timeline/waveformRenderer.ts)

• Refactored drawWaveform to accept a WaveformData object with both overview and detail peaks
• Renderer automatically uses detail peaks when they cover the visible viewport
• Falls back to overview peaks when detail is not available

Dynamic Waveform: Timeline Component (src/lib/components/Timeline.svelte)

• Added auto-fetch $effect that monitors zoom level and viewport
• When overview peaks per pixel drops below 2, triggers a 300ms-debounced detail extraction
• Detail extraction pads visible range by 50% on each side to avoid re-fetching on small pans
• Requests 4 peaks per pixel for crisp detail
• Passes WaveformData to both main timeline and minimap renderers

Renderer Types Updated (src/lib/timeline/renderer.ts)

• drawTimeline now accepts WaveformData instead of number[]

Lessons Learned

• WKWebView's custom protocol handlers (like Tauri's https://asset.localhost/) can have subtle audio issues even when video plays fine. Blob URLs are a reliable workaround.
• For waveform LOD (level of detail), a two-tier approach (high-count overview + on-demand range extraction) provides the best UX: fast initial display with detail on demand.
• Debouncing the detail waveform fetch prevents spamming ffmpeg during rapid zoom/pan.
• Padding the extraction range by 50% on each side significantly reduces re-fetch frequency during small viewport adjustments.

Follow-Up Items

• Investigate if Tauri's asset protocol can be configured to properly serve audio (may be a wry bug)
• Consider pre-computing multiple LOD levels for the waveform instead of on-demand extraction
• The blob approach loads the entire preview file into memory (~97MB for a 54-min video at 360p) — acceptable for desktop but may need optimization for very long videos