73 lines
4.5 KiB
Markdown
73 lines
4.5 KiB
Markdown
|
# Fix Audio Playback + Dynamic Waveform Resolution
|
||
|
|
|
||
|
|
## Task Description
|
||
|
|
1. **Fix missing audio on preview playback** — Video plays but with no sound in WKWebView.
|
||
|
|
2. **Dynamic waveform resolution** — Waveform should increase in detail as the user zooms in.
|
||
|
|
|
||
|
|
## Root Cause Analysis (Audio)
|
||
|
|
|
||
|
|
The preview file is correctly muxed (H.264 640x360 + AAC 128kbps, moov atom at offset 24 — fast-start). WKWebView config analysis:
|
||
|
|
- wry 0.55.1 defaults `autoplay: true` → sets `mediaTypesRequiringUserActionForPlayback = None`
|
||
|
|
- tauri-runtime-wry uses `WebViewBuilder::new_with_web_context()` which inherits this default
|
||
|
|
- Tauri v2.11.6 doesn't expose or override `autoplay`
|
||
|
|
- So WKWebView SHOULD be configured for audio playback
|
||
|
|
|
||
|
|
Despite correct configuration, WKWebView's Tauri asset protocol (`https://asset.localhost/...`) appears to silently drop audio tracks when streaming local files — possibly due to range request handling or MIME type issues in the custom protocol handler.
|
||
|
|
|
||
|
|
**Fix**: Load the video file as a `blob:` URL instead of using the asset protocol. This bypasses the asset protocol entirely and uses WKWebView's native blob URL handling, which reliably plays both audio and video tracks.
|
||
|
|
|
||
|
|
## Changes Made
|
||
|
|
|
||
|
|
### Audio Fix: Blob URL Video Loading (`src/lib/components/VideoPlayer.svelte`)
|
||
|
|
- Added a `$effect` that fetches the preview file via `convertFileSrc` URL, converts it to a Blob, then creates a `blob:` URL
|
||
|
|
- Video element now uses the blob URL instead of the asset protocol URL
|
||
|
|
- Added `playsinline` attribute to the video element
|
||
|
|
- Added explicit `volume = 1` and `muted = false` on `loadeddata`
|
||
|
|
- Added `.play()` Promise error handling (catches and logs rejections)
|
||
|
|
- Shows "Preparing video…" state while blob is loading
|
||
|
|
- Falls back to asset protocol URL if blob creation fails
|
||
|
|
- Properly revokes old blob URLs on session change
|
||
|
|
|
||
|
|
### Dynamic Waveform: Backend (`src-tauri/src/services/waveform_generator.rs`)
|
||
|
|
- Added `extract_waveform_range(file_path, start_time, end_time, peak_count)` function
|
||
|
|
- Uses `ffmpeg -ss {start} -t {duration}` with 44100 Hz sample rate for high-resolution extraction
|
||
|
|
- Computes peaks for just the specified time range
|
||
|
|
|
||
|
|
### Dynamic Waveform: Command (`src-tauri/src/commands/media_analysis.rs`)
|
||
|
|
- Added `extract_waveform_range` Tauri command
|
||
|
|
- Registered in `lib.rs`
|
||
|
|
|
||
|
|
### Dynamic Waveform: Frontend Binding (`src/lib/bindings/mediaAnalysis.ts`)
|
||
|
|
- Added `extractWaveformRange(filePath, startTime, endTime, peakCount)` binding
|
||
|
|
|
||
|
|
### Dynamic Waveform: Session State (`src/lib/stores/videoSession.svelte.ts`)
|
||
|
|
- Added `waveformDetailPeaks`, `waveformDetailStart`, `waveformDetailEnd` to session state
|
||
|
|
- Increased initial waveform extraction from 8,000 to 50,000 peaks (good for most zoom levels)
|
||
|
|
- Detail fields cleared on session reset
|
||
|
|
|
||
|
|
### Dynamic Waveform: Renderer (`src/lib/timeline/waveformRenderer.ts`)
|
||
|
|
- Refactored `drawWaveform` to accept a `WaveformData` object with both overview and detail peaks
|
||
|
|
- Renderer automatically uses detail peaks when they cover the visible viewport
|
||
|
|
- Falls back to overview peaks when detail is not available
|
||
|
|
|
||
|
|
### Dynamic Waveform: Timeline Component (`src/lib/components/Timeline.svelte`)
|
||
|
|
- Added auto-fetch `$effect` that monitors zoom level and viewport
|
||
|
|
- When overview peaks per pixel drops below 2, triggers a 300ms-debounced detail extraction
|
||
|
|
- Detail extraction pads visible range by 50% on each side to avoid re-fetching on small pans
|
||
|
|
- Requests 4 peaks per pixel for crisp detail
|
||
|
|
- Passes `WaveformData` to both main timeline and minimap renderers
|
||
|
|
|
||
|
|
### Renderer Types Updated (`src/lib/timeline/renderer.ts`)
|
||
|
|
- `drawTimeline` now accepts `WaveformData` instead of `number[]`
|
||
|
|
|
||
|
|
## Lessons Learned
|
||
|
|
- WKWebView's custom protocol handlers (like Tauri's `https://asset.localhost/`) can have subtle audio issues even when video plays fine. Blob URLs are a reliable workaround.
|
||
|
|
- For waveform LOD (level of detail), a two-tier approach (high-count overview + on-demand range extraction) provides the best UX: fast initial display with detail on demand.
|
||
|
|
- Debouncing the detail waveform fetch prevents spamming ffmpeg during rapid zoom/pan.
|
||
|
|
- Padding the extraction range by 50% on each side significantly reduces re-fetch frequency during small viewport adjustments.
|
||
|
|
|
||
|
|
## Follow-Up Items
|
||
|
|
- Investigate if Tauri's asset protocol can be configured to properly serve audio (may be a wry bug)
|
||
|
|
- Consider pre-computing multiple LOD levels for the waveform instead of on-demand extraction
|
||
|
|
- The blob approach loads the entire preview file into memory (~97MB for a 54-min video at 360p) — acceptable for desktop but may need optimization for very long videos
|