pi-weekly-newspaper/docs/superpowers/specs/2026-04-06-pi-weekly-newspaper-design.md

# Plymouth Independent Weekly Newspaper — Design Spec

## Goal

Publish a weekly ePub "newspaper" containing articles from the Plymouth Independent RSS feed, optimized for reading on an Xtreink X4 e-reader.

## Requirements Summary

- **Output:** ePub with articles as chapters, chronological order (Monday–Sunday ISO weeks)
- **Offline:** All images downloaded and embedded
- **E-reader formatting:** Images fit within 800x480 (landscape) or 480x800 (portrait) bounding box, aspect ratio preserved, baseline JPEG
- **Interface:** Self-hosted Python web app, accessible via browser from MacBook and Android phone on local network
- **Pipeline:** Periodic RSS fetch/cache, then manual or scheduled compile-and-publish
- **Cover:** AI-generated via Pollinations.ai (primary), programmatic text fallback, selectable at publish time
- **Article selection:** All articles included by default; user can exclude specific ones via UI before publishing

---

## Architecture

### Stack

| Component | Choice | Rationale |
|---|---|---|
| Web framework | Flask + Jinja2 | Lightweight, single-process |
| ORM / DB | Flask-SQLAlchemy + SQLite | Zero-config, single-file DB |
| Scheduler | APScheduler (BackgroundScheduler) | In-process, no external dependencies |
| RSS parsing | feedparser | Standard Python RSS library |
| ePub generation | ebooklib | Mature ePub 3 library |
| Image processing | Pillow | Resize, format conversion, text rendering |
| HTML parsing | beautifulsoup4 | Extract images from article HTML |
| HTTP | requests | Feed + image downloads |
| AI cover | Pollinations.ai | Free, no API key, URL-based |
| Frontend | Plain HTML + Pico CSS + vanilla JS | No build step, mobile-friendly |

### Project Structure

```
pi-weekly-newspaper/
├── app.py                  # Entry point: Flask app + APScheduler setup
├── config.py               # Config (feed URL, check interval, image dims, etc.)
├── requirements.txt
├── src/
│   ├── __init__.py
│   ├── fetcher.py          # RSS fetch, parse, cache articles to DB
│   ├── images.py           # Download images, resize, baseline JPEG conversion
│   ├── epub_builder.py     # Assemble ePub from cached articles + images
│   ├── cover.py            # Cover generation (Pollinations.ai + text fallback)
│   ├── models.py           # SQLAlchemy models (Article, Image, Issue, Settings)
│   └── scheduler.py        # APScheduler config, job management
├── static/                 # CSS, JS for web UI
├── templates/              # Jinja2 templates for web UI
├── data/
│   ├── newspaper.db        # SQLite database (created at runtime)
│   ├── images/             # Downloaded/processed images (runtime)
│   └── issues/             # Generated ePub files (runtime)
└── README.md
```

### Data Flow

1. **Fetch job** (periodic, default every 1 hour): RSS feed → parse → store new articles + metadata in SQLite → download & process images to `data/images/`
2. **Publish action** (manual via UI, or auto-scheduled): query articles for target week → user reviews/excludes via UI → generate cover → assemble ePub → save to `data/issues/` → download link available

---

## Data Model

### `articles`

| Column | Type | Notes |
|---|---|---|
| `id` | INTEGER PK | Auto-increment |
| `guid` | TEXT UNIQUE | RSS `<guid>`, deduplication key |
| `title` | TEXT | Article title |
| `author` | TEXT | `dc:creator` value |
| `pub_date` | DATETIME | Publication timestamp |
| `categories` | TEXT | JSON array of category strings |
| `link` | TEXT | Original article URL |
| `content_html` | TEXT | Full `content:encoded` HTML with local image refs |
| `fetched_at` | DATETIME | When we cached it |

### `images`

| Column | Type | Notes |
|---|---|---|
| `id` | INTEGER PK | Auto-increment |
| `article_id` | INTEGER FK | References `articles.id` |
| `original_url` | TEXT | Source URL from the article HTML |
| `local_path` | TEXT | Path to processed file in `data/images/` |
| `width` | INTEGER | Final width after resize |
| `height` | INTEGER | Final height after resize |

### `issues`

| Column | Type | Notes |
|---|---|---|
| `id` | INTEGER PK | Auto-increment |
| `week_start` | DATE | Monday of the ISO week |
| `week_end` | DATE | Sunday of the ISO week |
| `cover_method` | TEXT | `"ai"` or `"text"` |
| `cover_path` | TEXT | Path to cover image |
| `epub_path` | TEXT | Path to generated `.epub` |
| `article_ids` | TEXT | JSON array of included article IDs |
| `excluded_article_ids` | TEXT | JSON array of excluded article IDs |
| `created_at` | DATETIME | When the issue was generated |
| `status` | TEXT | `"draft"` / `"published"` |

### `settings`

| Column | Type | Notes |
|---|---|---|
| `key` | TEXT PK | Setting name |
| `value` | TEXT | JSON-encoded value |

Used for: feed URL, fetch interval, auto-publish config, image constraints. Read on startup to restore scheduler state.

---

## Module Details

### `fetcher.py` — RSS Fetch & Article Caching

1. Fetch RSS feed via `feedparser` + `requests`
2. Deduplicate by `guid` — skip articles already in DB
3. Parse each new `<item>`: title, author, pub_date, categories, link, content_html
4. **Save article record to SQLite first** (to obtain `article_id`)
5. Extract image URLs from `content:encoded` HTML using `BeautifulSoup` with `html.parser`
6. Download & process each image via `images.py` — store to `data/images/{url_hash}.jpg` (deduped by URL hash across all articles)
7. Create `images` DB records linking `article_id` to each processed image
8. Rewrite `<img src>` attributes in stored `content_html` to point to local paths
9. Update the article record with the rewritten `content_html`

**Edge cases:**
- Feed unavailable: log warning, retry next cycle, no crash
- Duplicate images across articles (same URL): download once, reference by URL hash
- Images that 404: log warning, skip image, article still included
- Malformed HTML: `BeautifulSoup` with `html.parser` is tolerant

### `images.py` — Image Processing

1. Download image from URL via `requests`
2. Check if `data/images/{url_hash}.jpg` already exists — if so, return cached path (dedup)
3. Open with Pillow
4. Determine orientation: if width >= height → landscape bounding box (800x480), else portrait (480x800)
5. Resize to fit within bounding box, preserving aspect ratio:
   - If image is **larger** than the box: use `Image.thumbnail()` to scale down
   - If image is **smaller** than the box: use `Image.resize()` with `LANCZOS` to scale up, so it renders at a reasonable size on the e-reader
6. Save as baseline JPEG (`progressive=False`)
7. Return local path and final dimensions

### `epub_builder.py` — ePub Assembly

1. Query articles for target ISO week (Monday–Sunday), minus excluded ones
2. Sort chronologically by `pub_date`
3. Build ePub structure with `ebooklib`:
   - **Metadata:** title ("Plymouth Independent — Week of Apr 7–13, 2026"), language (en)
   - **Cover:** generated JPEG as ePub cover image
   - **Table of Contents:** article titles linked to chapters
   - **Chapters:** one per article, chronological
4. Each chapter:
   - `<h1>` article title
   - Author/date byline, category tags
   - Article HTML with image `src` rewritten to ePub-internal references
   - All referenced images embedded as ePub items
5. Stylesheet: minimal CSS for e-ink — no colors, high contrast, images `max-width: 100%; display: block`
6. Output: `data/issues/plymouth-independent-2026-W15.epub`

### `cover.py` — Cover Generation

**AI mode (Pollinations.ai):**
1. Build a prompt from the week's top headlines: "Newspaper front page illustration for Plymouth Massachusetts local news, featuring: [top 3 titles], classic newspaper style"
2. Fetch from `https://image.pollinations.ai/prompt/{encoded_prompt}?width=800&height=480`
3. Resize/fit to 800x480 bounding box, baseline JPEG
4. Overlay masthead text ("Plymouth Independent") and date range using Pillow `ImageDraw`

**Text fallback mode:**
1. Create 800x480 Pillow image with white background
2. Draw bold "Plymouth Independent" masthead
3. Date range subtitle
4. List top article headlines
5. Save as baseline JPEG

Both modes produce a single baseline JPEG within e-reader constraints.

### `scheduler.py` — Background Scheduling

- APScheduler `BackgroundScheduler`, started on app launch
- Two jobs:
  1. **RSS fetch:** `IntervalTrigger`, default every 1 hour
  2. **Auto-publish** (optional): `CronTrigger`, configurable day/time
- Schedule config persisted to `settings` table in SQLite
- On startup: read settings from DB, restore scheduler jobs
- Web UI can pause/resume/reconfigure jobs live

---

## Web UI

Five views, all server-rendered with Jinja2. Responsive layout via Pico CSS.

### Dashboard (`/`)
- Scheduler status (running/paused, next fetch, interval)
- Quick stats: articles this week, total cached, latest issue
- Buttons: "Fetch Now", "New Issue"

### Articles (`/articles`)
- Table of cached articles, filterable by week and category
- Columns: title, author, date, categories, thumbnail
- When preparing an issue: checkboxes for include/exclude

### Publish (`/publish`)
- Select target week (defaults to current ISO week)
- Article list with include/exclude toggles (all on by default)
- Cover method picker: "AI Cover" / "Text Cover"
- "Generate Issue" button
- Progress: synchronous POST request with a CSS spinner overlay; generation typically takes 5–15 seconds (dominated by Pollinations.ai round-trip if using AI cover)
- On completion: page reloads with download link and cover preview

### Settings (`/settings`)
- RSS feed URL
- Fetch interval (hours)
- Auto-publish: toggle + day/time + default cover method
- Image resize constraints

### Issues Archive (`/issues`)
- List of past issues: date range, article count, cover thumbnail
- Download link per issue
- "Regenerate" button

---

## Error Handling

| Scenario | Behavior |
|---|---|
| RSS feed down | Log warning, skip cycle, retry next interval |
| Image download fails | Log warning, skip image, include article without it |
| Pollinations.ai fails | Log error, fall back to text cover automatically |
| ePub generation fails | Show error in UI with details, don't save partial issue |
| DB locked (concurrent access) | SQLite WAL mode for better concurrency; scheduler and web requests share the same process |

---

## Future Enhancements (Out of Scope for V1)

- Full web scraping of article pages for richer content
- Email delivery of issues
- Multiple RSS feed support
- Reading progress tracking
- Dark mode cover variants