Files
cardputerzero-camera/docs/architecture-and-protocol.md

131 lines
9.3 KiB
Markdown
Raw Permalink Normal View History

> [!IMPORTANT]
> This project was developed entirely with AI coding assistance (Google Neon) and has not undergone rigorous review. It is provided as-is and may require adjustments for other environments.
# Architecture & Camera Protocol Reference
`czcamera` separates user-space UI, image processing, media assembly, and storage management in C++17 from Raspberry Pi hardware capture in Python 3 (`python3-picamera2`) through an explicit backend abstraction (`cz::ICameraBackend`).
---
## 1. System Topology & Decoupling Boundary
```text
+-------------------------------------------------------------------+
| C++17 /usr/bin/czcamera |
| - AppView (320x170 ShadcnCanvas: FreeType2 + 5x7 fallback) |
| - UiController, TimelapseEngine, GalleryStore, VideoPlayer |
| - Filters RGB24 + libjpeg still pipeline, BacklightController |
| - Linux (/dev/fb0 + cp0-keys evdev) OR Desktop SDL2 (--sim) |
| | |
| cz::ICameraBackend |
| +---------------+---------------+ |
| | | |
| SubprocessCameraBackend SimCameraBackend |
| (device / --mock-driver) (--sim / --mock-cam / tests) |
+--------------|----------------------------------------------------+
| stdin/stdout JSON v1 + dedicated CZFR fd pipe
v
+-------------------------------------------------------------------+
| Python 3 /usr/bin/czcamera-cam (czcamera_cam) |
| - JsonProtocolServer + FramePipeWriter (latest-slot drop-oldest) |
| - Picam2Worker (main + lores YUV420 dual streams, H264Encoder) |
| - MockCameraDriver (--mock-driver deterministic test driver) |
+-------------------------------------------------------------------+
| python3-picamera2 / libcamera IPA (rpi/vc4)
v
Sony IMX219 (3280x2464) + ES8389 MEMS Mic (ALSA default)
```
### Replacing `picamera2` with Native `libcamera` Later
`czcamera` C++ sources have **zero** `#include <libcamera/...>` or CMake linkage to `libcamera.so`. To swap `python3-picamera2` for a native C++ `libcamera` backend (or `rpicam-apps`), implement [`app/src/camera_backend.hpp`](../app/src/camera_backend.hpp) (`cz::ICameraBackend`) and return it from `create_camera_backend()` in [`app/src/sim_backend.cpp`](../app/src/sim_backend.cpp) without touching `UiController`, `AppView`, filters, timelapse, or gallery code.
---
## 2. Camera Helper Wire Protocol v1
`SubprocessCameraBackend` launches `/usr/bin/czcamera-cam --frame-fd <N>` (overridable via `CZCAMERA_CAM_BIN`, passing `--mock-driver` in protocol tests) with an inherited read/write pipe (`frame_fd`).
### Control Channel (`stdin` / `stdout` Newline-Delimited JSON)
Requests from C++ → Python carry `"id"` (uint64) and `"cmd"`:
| Command | Payload Fields | Reply Extra Fields (`{"id", "ok": true, ...}`) |
|---|---|---|
| `configure` | `mode` (`"photo"`/`"video"`/`"timelapse"`), `main_w`, `main_h`, `preview_w` (`226`), `preview_h` (`170`), `fps`, `zoom`, `ev`, `transform` (`"auto"`/`"0"`/`"180"`/`"hflip"`/`"vflip"`) | `mode`, `main_w`, `main_h`, `preview_w`, `preview_h` (`226x128` for 16:9 modes), `transform`, `is_16_9` |
| `set_controls` | Optional `zoom` (`1.0..4.0`), `ev` (`-2.0..+2.0`) | `zoom`, `ev` |
| `capture_still` | `tmp_path`, `width`, `height` | `bytes` (temporarily stops video / switches to still format if needed and restores stream config) |
| `start_video` | `tmp_path`, `width`, `height`, `fps`, `bitrate`, `audio` (`bool`) | `audio_recorded` (`bool`) |
| `stop_video` | — | `duration_sec`, `bytes`, `audio_recorded` |
| `pause` | — | Stops recording if active, closes camera & releases IMX219 lock |
| `resume` | — | Re-opens camera and restores last `configure` spec |
| `shutdown` | — | Tears down writer & camera cleanly and exits `0` |
Asynchronous Events from Python → C++ (no `"id"`):
- `{"event": "ready", "protocol": 1, "backend": "picamera2"|"mock", "sensor": "...", "has_audio": true|false}`
- `{"event": "warn", "code": "...", "message": "..."}` (for example `"audio_unavailable"`, `"invalid_json"`, `"control_error"`)
- `{"event": "error", "code": "...", "message": "..."}`
### Binary Preview Frame Channel (`--frame-fd <N>`, 16-Byte `CZFR` Header)
Every preview frame packet starts with a fixed **16-byte little-endian** header (`struct.Struct("<4sHHBBHI")`) followed immediately by `payload_len` bytes:
| Offset | Size | C Type | Field | Description |
|---|---|---|---|---|
| `0` | `4` | `char[4]` | `magic` | `"CZFR"` (`0x43 0x5A 0x46 0x52`) |
| `4` | `2` | `uint16_t` | `width` | Negotiated preview frame width (e.g., `226`) |
| `6` | `2` | `uint16_t` | `height` | Preview frame height (`170` for 4:3, `128` for 16:9) |
| `8` | `1` | `uint8_t` | `pixfmt` | `1` = `YUV420`, `2` = `RGB24` |
| `9` | `1` | `uint8_t` | `flags` | Bit `0` (`0x01`): `FLAG_ASPECT_16_9` |
| `10` | `2` | `uint16_t` | `reserved` | Always `0` |
| `12` | `4` | `uint32_t` | `payload_len` | `width * height * 3 / 2` (`YUV420`) or `width * height * 3` (`RGB24`) |
**Backpressure Contract (Review Focus #1):**
- `FramePipeWriter` uses a single-slot bounded queue (`_pending`). When the UI consumer slows down, `push_frame()` drops any unconsumed queued packet and replaces it with the newest frame, keeping camera/H.264 encoder threads non-blocking.
- `Picam2Worker._extract_lores_yuv420()` inspects the negotiated `lores` `stride` (handling 64-byte DMA padding such as `256` for `226px`) and strips row padding before packing tight `YUV420` payloads.
---
## 3. Target Hardware Constraints & Media Pipelines
- **Sony IMX219 Sensor Modes:** `640×480` (4:3, crop `(1000,752)/1280×960`), `1640×1232` (4:3 full-FOV 2×2 binned), `1920×1080` (16:9 crop), `3280×2464` (4:3 8MP full sensor).
- **Video Output (`FfmpegOutput`):** `_FfmpegOutputTarget` overrides `.split()` cleanly around `picamera2.outputs.FfmpegOutput`'s whitespace `.split()` so staging paths containing spaces or special characters (and ending in `.mp4.part`, with `-f mp4 -movflags +faststart`) remain exact `argv` tokens.
- **Timelapse Compilation:** [`TimelapseEngine`](../app/src/timelapse_engine.hpp) invokes `ffmpeg` with `-threads 1 -filter_threads 1` and scales `4:3` (`2mp`/`8mp`) to `1024:768` or `16:9` (`1080p`) to `1280:720` (`bilinear,setsar=1`) to stay safely within CardputerZero's 512 MB RAM / 64 MB CMA budget and VideoCore IV `/dev/video11` constraints, encoding with `h264_v4l2m2m` (`-num_output_buffers 4`, `5M`, `yuv420p`) and falling back to low-memory `libx264` (`-threads 2 -preset ultrafast -crf 23 -x264-params rc-lookahead=0:ref=1:bframes=0:threads=2:lookahead-threads=1:sync-lookahead=0`) on failure.
- **Gallery Playback:** [`VideoPlayer`](../app/src/video_player.hpp) probes duration & audio presence via `ffprobe`, spawns `ffmpeg` raw `rgb24` frames timed cleanly to PTS, and drives audio through ALSA `aplay` on Linux or `set_pcm_sink` (SDL2 audio queue) on desktop—allowing silent `.mp4` files (compiled timelapses or muted clips) to play smoothly without audio failure.
---
## 4. `320×170` UI Renderer & Zinc Color Tokens
[`AppView`](../app/src/app_ui.hpp) renders into an offscreen `320×170` ARGB8888 canvas with SDF antialiased rounded primitives and FreeType2 + `5×7` bitmap fallback, then blits via [`fb_blit.hpp`](../app/src/fb_blit.hpp) (`RGB565` / `RGB24` / `XRGB8888`) or SDL2 texture.
Exact palette tokens defined in [`app/src/theme.cpp`](../app/src/theme.cpp):
| Token | Zinc Dark (`ThemeMode::Dark`) | Zinc Light (`ThemeMode::Light`) |
|---|---|---|
| `background` / `card` | `#09090b` / `#09090b` | `#fafafa` / `#ffffff` |
| `muted` | `#18181b` | `#f4f4f5` |
| `border` | `#27272a` | `#e4e4e7` |
| `ring` | `#d4d4d8` | `#71717a` |
| `foreground` / `muted_foreground` | `#fafafa` / `#a1a1aa` | `#09090b` / `#71717a` |
| `active_tab` / `active_tab_foreground` | `#27272a` / `#fafafa` | `#ffffff` / `#09090b` |
| `status_online` / `status_online_bg` | `#4ade80` / `#11261a` | `#16a34a` / `#dcfce7` |
| `status_sleeping` / `status_sleeping_bg` | `#fbbf24` / `#271d0b` | `#d97706` / `#fef3c7` |
| `status_error` / `status_error_bg` | `#f87171` / `#2b1214` | `#dc2626` / `#fee2e2` |
| `viewfinder_surround` | `#050507` | `#050507` |
---
## 5. Source Directory Map
- **`app/src/`**:
- `theme.{hpp,cpp}`, `fb_blit.hpp`, `config.{hpp,cpp}`, `storage.{hpp,cpp}`
- `filters.{hpp,cpp}`, `jpeg_pipeline.{hpp,cpp}`
- `camera_backend.hpp`, `subprocess_backend.{hpp,cpp}`, `sim_backend.{hpp,cpp}`
- `backlight.{hpp,cpp}`, `timelapse_engine.{hpp,cpp}`, `gallery_model.{hpp,cpp}`, `video_player.{hpp,cpp}`
- `ui_controller.{hpp,cpp}`, `app_ui.{hpp,cpp}`, `evdev_repeat_guard.hpp`, `main.cpp`
- **`app/tests/`**: `test_config_and_storage.cpp`, `test_filters_and_jpeg.cpp`, `test_backend_protocol.cpp`, `test_timelapse_and_gallery.cpp`, `test_ui_controller.cpp`
- **`cam-helper/`**: `bin/czcamera-cam`, `czcamera_cam/{__init__,protocol,picam2_worker}.py`, `tests/test_{protocol,picam2_worker}.py`
- **`packaging/`**: `Dockerfile.arm64`, `build-deb.sh`, `czcamera.desktop`, `czcamera.png`, `debian/{control,postinst,prerm}`, `tests/test_packaging.py`
- **`docs/`**: User/developer guides (`building-and-installation.md`, `usage-and-controls.md`, `architecture-and-protocol.md`)
- **`LICENSE.md`**: Apache License, Version 2.0