feat: integrated epub optimizer (#1224)

## Problem

Many e-ink readers have limited image decoder support natively.
EPUBs with images in other formats than **baseline JPEG** frequently
cause:

- **Broken images**: pages render as blank, corrupted noise, or never
load
- **Slow rendering**: unoptimized images cause severe delays on e-ink
hardware, up to 7 seconds per page turn, with cover images taking up to
59 seconds to render
- **Broken covers**: the book thumbnail never generates

Fixing this today requires external tools before uploading.

---

## What this PR does

Adds an **optional, on-demand EPUB optimizer** to the file upload flow.
When enabled,
it converts all images to baseline JPEG directly in the browser — no
server, no internet,
no external tools needed.

**Conversion is opt-in. The standard upload flow is unchanged.**

---

## Real-world impact

The optimizer was applied in batch to **61 EPUBs**:
- 60 standard EPUBs: 198 MB → 55 MB (**−72.2%**, 143 MB saved)
- Text-dominant books: 8–46% smaller (covers and inline images
converted)
  - Image-heavy / illustrated books: 65–93% smaller
- 1 Large manga volume (594 MB): 594 MB → 72 MB (**−87.8%**, 522 MB
saved)
- EPUB structural integrity fully maintained — zero new validation
issues introduced across all 61 books

*Size and integrity analysis:
[epub-comparator](https://github.com/pablohc/epub-comparator)*

From that set, **17 books were selected** as a representative sample
covering different content
types: image-heavy novels, pure manga, light novels with broken images,
and text-dominant books.
Each was benchmarked on two devices running in parallel, one on `master`
and one
on `PR#1224` — measuring render time across ~30 pages per book on
average.

### Rendering bugs fixed

| Book | Problem (original) | After optimization |
|------|--------------------|--------------------|
| Fairy Tale — Stephen King | Cover took **59.7 s** to render | 2.1 s
(−96%) |
| Cycle of the Werewolf — Stephen King | Cover took **23.3 s** to render
| 1.7 s (−93%) |
| Tomie: Complete Deluxe Ed. — Junji Ito | Cover took **18.3 s** to
render | 2.0 s (−89%) |
| Joel Dicker — El tigre (Ed. Ilustrada) | Cover took **14.5 s** to
render | 1.4 s (−90%) |
| Jackson, Holly — Asesinato para principiantes | Cover failed
completely (blank) | 2.0 s ✓ |
| Sentenced to Be a Hero — Yen Press | Cover failed, **8 images failed
to load** | All fixed ✓ |
| Flynn, Gillian — Perdida | Cover failed completely (blank) | 1.6 s ✓ |
| Chandler, Raymond — Asesino en la lluvia | Cover failed completely
(blank) | 2.0 s ✓ |

### Page render times — image-heavy EPUBs (avg per page)

| Book | Pages | Avg original | Avg optimized | Improvement | File size
|
|------|-------|-------------|---------------|-------------|-----------|
| Fairy Tale — Stephen King | 30 | 3,028 ms | 1,066 ms | **−64.8%** |
32.4 MB → 9.1 MB (−72%) |
| Cycle of the Werewolf — Stephen King | 33 | 3,026 ms | 1,558 ms |
**−48.5%** | 35.1 MB → 2.9 MB (−92%) |
| Joel Dicker — El tigre (Ed. Ilustrada) | 16 | 1,846 ms | 1,051 ms |
**−43.1%** | 5.3 MB → 0.4 MB (−93%) |
| Tomie: Complete Deluxe Ed. — Junji Ito | 30 | 4,817 ms | 2,802 ms |
**−41.8%** | 593.8 MB → 72.2 MB (−87.8%) |
| Sentenced to Be a Hero — Yen Press | 30 | 1,719 ms | 1,388 ms |
**−19.2%** | 15.2 MB → 1.6 MB (−90%) |

### Text-heavy EPUBs — no regression

| Book | Pages | Avg original | Avg optimized | Delta |
|------|-------|-------------|---------------|-------|
| Christie — Asesinato en el Orient Express | 30 | 1,672 ms | 1,646 ms |
−1.6% |
| Flynn — Perdida | 30 | 1,327 ms | 1,291 ms | −2.7% |
| Dicker — La verdad sobre el caso Harry Quebert | 30 | 1,132 ms | 1,084
ms | −4.2% |
| Hammett — El halcón maltés | 30 | 1,009 ms | 966 ms | −4.3% |
| Chandler — Asesino en la lluvia | 30 | 989 ms | 1,007 ms | +1.8% |

*Differences within ±5% — consistent with device measurement noise.*

*Render time benchmark:
[epub-optimization-benchmark](https://github.com/pablohc/epub-optimization-benchmark)*

---
## How to use it

**Single file:**
1. Click **Upload** (top of the page) — a modal opens. Use **Choose
files** to select one EPUB from your device.
2. Check **Optimize**.
- *(Optional)* Expand **Advanced Mode** — adjust quality, rotation, or
overlap; set individual images to H-Split / V-Split / Rotate.
3. Click **Optimize & Upload**.

**Batch (2+ files):**
1. Click **Upload** (top of the page) — a modal opens. Use **Choose
files** to select multiple EPUBs from your device.
2. Check **Optimize**.
   - *(Optional)* Expand **Advanced Mode** — adjust quality.
3. Click **Upload** — all files are converted and uploaded sequentially.

Upload a batch of files, without optimization:
<img width="810" height="671" alt="image"
src="https://github.com/user-attachments/assets/d892ae13-0b87-4ea4-b6b8-340d56efc763"
/>

Batch file upload, with standard optimization:
<img width="809" height="707" alt="image"
src="https://github.com/user-attachments/assets/d32dbc88-1208-4555-bfcf-330ab91d2174"
/>

Optimization Phase (1/2):
<img width="807" height="1055" alt="image"
src="https://github.com/user-attachments/assets/fd4cd5f9-e56e-4ca1-9777-6926b9baf2bb"
/>

Upload Phase (2/2):
<img width="805" height="1065" alt="image"
src="https://github.com/user-attachments/assets/483294f0-02f0-4569-ae11-c10b3581d747"
/>

Batch upload successfully confirmed:
<img width="812" height="1043" alt="image"
src="https://github.com/user-attachments/assets/80c135bf-05c3-4c80-8755-2a04c68235bc"
/>

---

## Options

**Always active when the converter is enabled:**
- Converts PNG, WebP, BMP, GIF → baseline JPEG
- Smart downscaling to 480×800 px max (preserves aspect ratio)
- True grayscale for e-ink (BT.709 luminance, always on)
- SVG cover fix + OPF/NCX compliance repairs

**Advanced Mode (opt-in) — single file:**
- JPEG quality presets: 30% / 45% / 60% / 75% / **85%** (default) / 95%
- Rotation direction for split images: CW (default) / CCW
- Min overlap when splitting: 5% (default) / 10% / 15%
- Auto-download conversion log toggle (detailed stats per image)
- Per-image picker: set Normal / H-Split / V-Split / Rotate per image
individually,
  with "Apply to all" for bulk assignment

**Advanced Mode (opt-in) — batch (2+ files):**
- JPEG quality presets: 30% / 45% / 60% / 75% / **85%** (default) / 95%
- Auto-download conversion log toggle (aggregated stats for all files)

---

## ⚠️ Known limitations

**KoReader hash-based sync will break** for converted files. The file
content changes,
so the hash no longer matches the original. Filename-based sync is
unaffected.

If you rely on KoReader hash sync, use the Calibre plugin or the web
tool instead.

---
## Build size impact

| Metric | master (53beeee) | PR #1224 (a2ba5db) | Delta |

|---------------|------------------|--------------------|----------------|
| Flash used | 5,557 KB | 5,616 KB | +59 KB (+1.1%) |
| Flash free | 843 KB | 784 KB | −59 KB |
| Flash usage | 86.8% | 87.7% | +0.9 pp |
| RAM used | 95,156 B | 95,156 B | no change |

> Both builds compiled with `gh_release` environment in release mode
(ESP32-C3, 6,400 KB Flash).
> The +59 KB increase is entirely due to `jszip.min.js` embedded as a
> gzipped static asset served from Flash. RAM usage is identical,
> confirming no runtime overhead — the library runs in the browser,
> not on the ESP32. ~784 KB of Flash remain available.

---

## Alternatives considered

| Approach | Friction |
|----------|---------|
| **This PR** — integrated in upload flow | Zero: convert + upload in
one step, offline, any browser |
| Calibre plugin (in parallel development) | Requires a computer with
Calibre installed, same network |
| Web converters | Requires extra upload / download / transfer steps |

---

## Credits

Based on the converter algorithm developed by @zgredex.
Co-authored-by: @zgredex

---

### AI Usage

Did you use AI tools to help write this code? **PARTIALLY**

---------

Co-authored-by: zgredex <zgredex@users.noreply.github.com>

scripts/build_html.py

@@ -32,21 +32,41 @@ def minify_html(html: str) -> str:
 
     return html.strip()
 
+def sanitize_identifier(name: str) -> str:
+    """Sanitize a filename to create a valid C identifier.
+
+    C identifiers must:
+    - Start with a letter or underscore
+    - Contain only letters, digits, and underscores
+    """
+    # Replace non-alphanumeric characters (including hyphens) with underscores
+    sanitized = re.sub(r'[^a-zA-Z0-9_]', '_', name)
+    # Prefix with underscore if starts with a digit
+    if sanitized and sanitized[0].isdigit():
+        sanitized = f"_{sanitized}"
+    return sanitized
+
 for root, _, files in os.walk(SRC_DIR):
     for file in files:
-        if file.endswith(".html"):
+        if file.endswith(".html") or file.endswith(".js"):
-            html_path = os.path.join(root, file)
+            file_path = os.path.join(root, file)
-            with open(html_path, "r", encoding="utf-8") as f:
+            with open(file_path, "r", encoding="utf-8") as f:
-                html_content = f.read()
+                content = f.read()
 
-            # minified = regex.sub("\g<1>", html_content)
+            # Only minify HTML files; JS files are typically pre-minified (e.g., jszip.min.js)
-            minified = minify_html(html_content)
+            if file.endswith(".html"):
+                processed = minify_html(content)
+            else:
+                processed = content
 
             # Compress with gzip (compresslevel 9 is maximum compression)
             # IMPORTANT: we don't use brotli because Firefox doesn't support brotli with insecured context (only supported on HTTPS)
-            compressed = gzip.compress(minified.encode('utf-8'), compresslevel=9)
+            compressed = gzip.compress(processed.encode('utf-8'), compresslevel=9)
 
-            base_name = f"{os.path.splitext(file)[0]}Html"
+            # Create valid C identifier from filename
+            # Use appropriate suffix based on file type
+            suffix = "Html" if file.endswith(".html") else "Js"
+            base_name = sanitize_identifier(f"{os.path.splitext(file)[0]}{suffix}")
             header_path = os.path.join(root, f"{base_name}.generated.h")
 
             with open(header_path, "w", encoding="utf-8") as h:
@@ -65,10 +85,9 @@ for root, _, files in os.walk(SRC_DIR):
 
                 h.write(f"}};\n\n")
                 h.write(f"constexpr size_t {base_name}CompressedSize = {len(compressed)};\n")
-                h.write(f"constexpr size_t {base_name}OriginalSize = {len(minified)};\n")
+                h.write(f"constexpr size_t {base_name}OriginalSize = {len(processed)};\n")
 
             print(f"Generated: {header_path}")
-            print(f"  Original: {len(html_content)} bytes")
+            print(f"  Original: {len(content)} bytes")
-            print(f"  Minified: {len(minified)} bytes ({100*len(minified)/len(html_content):.1f}%)")
+            print(f"  Minified: {len(processed)} bytes ({100*len(processed)/len(content):.1f}%)")
-            print(f"  Compressed: {len(compressed)} bytes ({100*len(compressed)/len(html_content):.1f}%)")
+            print(f"  Compressed: {len(compressed)} bytes ({100*len(compressed)/len(content):.1f}%)")
-

src/network/CrossPointWebServer.cpp

@@ -16,6 +16,7 @@
 #include "html/FilesPageHtml.generated.h"
 #include "html/HomePageHtml.generated.h"
 #include "html/SettingsPageHtml.generated.h"
+#include "html/js/jszip_minJs.generated.h"
 
 namespace {
 // Folders/files to hide from the web interface file browser
@@ -36,6 +37,8 @@ size_t wsUploadSize = 0;
 size_t wsUploadReceived = 0;
 unsigned long wsUploadStartTime = 0;
 bool wsUploadInProgress = false;
+uint8_t wsUploadClientNum = 255;  // 255 = no active upload client
+size_t wsLastProgressSent = 0;
 String wsLastCompleteName;
 size_t wsLastCompleteSize = 0;
 unsigned long wsLastCompleteAt = 0;
@@ -131,6 +134,7 @@ void CrossPointWebServer::begin() {
   LOG_DBG("WEB", "Setting up routes...");
   server->on("/", HTTP_GET, [this] { handleRoot(); });
   server->on("/files", HTTP_GET, [this] { handleFileList(); });
+  server->on("/js/jszip.min.js", HTTP_GET, [this] { handleJszip(); });
 
   server->on("/api/status", HTTP_GET, [this] { handleStatus(); });
   server->on("/api/files", HTTP_GET, [this] { handleFileListData(); });
@@ -188,6 +192,21 @@ void CrossPointWebServer::begin() {
   LOG_DBG("WEB", "[MEM] Free heap after server.begin(): %d bytes", ESP.getFreeHeap());
 }
 
+void CrossPointWebServer::abortWsUpload(const char* tag) {
+  wsUploadFile.close();
+  String filePath = wsUploadPath;
+  if (!filePath.endsWith("/")) filePath += "/";
+  filePath += wsUploadFileName;
+  if (Storage.remove(filePath.c_str())) {
+    LOG_DBG(tag, "Deleted incomplete upload: %s", filePath.c_str());
+  } else {
+    LOG_DBG(tag, "Failed to delete incomplete upload: %s", filePath.c_str());
+  }
+  wsUploadInProgress = false;
+  wsUploadClientNum = 255;
+  wsLastProgressSent = 0;
+}
+
 void CrossPointWebServer::stop() {
   if (!running || !server) {
     LOG_DBG("WEB", "stop() called but already stopped (running=%d, server=%p)", running, server.get());
@@ -199,10 +218,9 @@ void CrossPointWebServer::stop() {
 
   LOG_DBG("WEB", "[MEM] Free heap before stop: %d bytes", ESP.getFreeHeap());
 
-  // Close any in-progress WebSocket upload
+  // Close any in-progress WebSocket upload and remove partial file
   if (wsUploadInProgress && wsUploadFile) {
-    wsUploadFile.close();
+    abortWsUpload("WEB");
-    wsUploadInProgress = false;
   }
 
   // Stop WebSocket server
@@ -309,6 +327,12 @@ void CrossPointWebServer::handleRoot() const {
   LOG_DBG("WEB", "Served root page");
 }
 
+void CrossPointWebServer::handleJszip() const {
+  server->sendHeader("Content-Encoding", "gzip");
+  server->send_P(200, "application/javascript", jszip_minJs, jszip_minJsCompressedSize);
+  LOG_DBG("WEB", "Served jszip.min.js");
+}
+
 void CrossPointWebServer::handleNotFound() const {
   String message = "404 Not Found\n\n";
   message += "URI: " + server->uri() + "\n";
@@ -505,7 +529,26 @@ void CrossPointWebServer::handleDownload() const {
   server->send(200, contentType.c_str(), "");
 
   NetworkClient client = server->client();
-  client.write(file);
+  const size_t chunkSize = 4096;
+  uint8_t buffer[chunkSize];
+
+  bool downloadOk = true;
+  while (downloadOk && file.available()) {
+    int result = file.read(buffer, chunkSize);
+    if (result <= 0) break;
+    size_t bytesRead = static_cast<size_t>(result);
+    size_t totalWritten = 0;
+    while (totalWritten < bytesRead) {
+      esp_task_wdt_reset();
+      size_t wrote = client.write(buffer + totalWritten, bytesRead - totalWritten);
+      if (wrote == 0) {
+        downloadOk = false;
+        break;
+      }
+      totalWritten += wrote;
+    }
+  }
+  client.clear();
   file.close();
 }
 
@@ -1082,7 +1125,7 @@ void CrossPointWebServer::handleGetSettings() const {
         doc["type"] = "string";
         if (s.stringGetter) {
           doc["value"] = s.stringGetter();
-        } else if (s.stringOffset > 0) {
+        } else if (s.stringMaxLen > 0) {
           doc["value"] = reinterpret_cast<const char*>(&SETTINGS) + s.stringOffset;
         }
         break;
@@ -1166,7 +1209,7 @@ void CrossPointWebServer::handlePostSettings() {
         const std::string val = doc[s.key].as<std::string>();
         if (s.stringSetter) {
           s.stringSetter(val);
-        } else if (s.stringOffset > 0 && s.stringMaxLen > 0) {
+        } else if (s.stringMaxLen > 0) {
           char* ptr = reinterpret_cast<char*>(&SETTINGS) + s.stringOffset;
           strncpy(ptr, val.c_str(), s.stringMaxLen - 1);
           ptr[s.stringMaxLen - 1] = '\0';
@@ -1202,17 +1245,12 @@ void CrossPointWebServer::onWebSocketEvent(uint8_t num, WStype_t type, uint8_t*
   switch (type) {
     case WStype_DISCONNECTED:
       LOG_DBG("WS", "Client %u disconnected", num);
-      // Clean up any in-progress upload
+      // Only clean up if this is the client that owns the active upload.
-      if (wsUploadInProgress && wsUploadFile) {
+      // A new client may have already started a fresh upload before this
-        wsUploadFile.close();
+      // DISCONNECTED event fires (race condition on quick cancel + retry).
-        // Delete incomplete file
+      if (num == wsUploadClientNum && wsUploadInProgress && wsUploadFile) {
-        String filePath = wsUploadPath;
+        abortWsUpload("WS");
-        if (!filePath.endsWith("/")) filePath += "/";
-        filePath += wsUploadFileName;
-        Storage.remove(filePath.c_str());
-        LOG_DBG("WS", "Deleted incomplete upload: %s", filePath.c_str());
       }
-      wsUploadInProgress = false;
       break;
 
     case WStype_CONNECTED: {
@@ -1226,15 +1264,35 @@ void CrossPointWebServer::onWebSocketEvent(uint8_t num, WStype_t type, uint8_t*
       LOG_DBG("WS", "Text from client %u: %s", num, msg.c_str());
 
       if (msg.startsWith("START:")) {
+        // Reject any START while an upload is already active to prevent
+        // leaking the open wsUploadFile handle (owning client re-START included)
+        if (wsUploadInProgress) {
+          wsServer->sendTXT(num, "ERROR:Upload already in progress");
+          break;
+        }
+
         // Parse: START:<filename>:<size>:<path>
         int firstColon = msg.indexOf(':', 6);
         int secondColon = msg.indexOf(':', firstColon + 1);
 
         if (firstColon > 0 && secondColon > 0) {
           wsUploadFileName = msg.substring(6, firstColon);
-          wsUploadSize = msg.substring(firstColon + 1, secondColon).toInt();
+          String sizeToken = msg.substring(firstColon + 1, secondColon);
+          bool sizeValid = sizeToken.length() > 0;
+          int digitStart = (sizeValid && sizeToken[0] == '+') ? 1 : 0;
+          if (digitStart > 0 && sizeToken.length() < 2) sizeValid = false;
+          for (int i = digitStart; i < (int)sizeToken.length() && sizeValid; i++) {
+            if (!isdigit((unsigned char)sizeToken[i])) sizeValid = false;
+          }
+          if (!sizeValid) {
+            LOG_DBG("WS", "START rejected: invalid size token '%s'", sizeToken.c_str());
+            wsServer->sendTXT(num, "ERROR:Invalid START format");
+            return;
+          }
+          wsUploadSize = sizeToken.toInt();
           wsUploadPath = msg.substring(secondColon + 1);
           wsUploadReceived = 0;
+          wsLastProgressSent = 0;
           wsUploadStartTime = millis();
 
           // Ensure path is valid
@@ -1262,10 +1320,25 @@ void CrossPointWebServer::onWebSocketEvent(uint8_t num, WStype_t type, uint8_t*
           if (!Storage.openFileForWrite("WS", filePath, wsUploadFile)) {
             wsServer->sendTXT(num, "ERROR:Failed to create file");
             wsUploadInProgress = false;
+            wsUploadClientNum = 255;
             return;
           }
           esp_task_wdt_reset();
 
+          // Zero-byte upload: complete immediately without waiting for BIN frames
+          if (wsUploadSize == 0) {
+            wsUploadFile.close();
+            wsLastCompleteName = wsUploadFileName;
+            wsLastCompleteSize = 0;
+            wsLastCompleteAt = millis();
+            LOG_DBG("WS", "Zero-byte upload complete: %s", filePath.c_str());
+            clearEpubCacheIfNeeded(filePath);
+            wsServer->sendTXT(num, "DONE");
+            wsLastProgressSent = 0;
+            break;
+          }
+
+          wsUploadClientNum = num;
           wsUploadInProgress = true;
           wsServer->sendTXT(num, "READY");
         } else {
@@ -1276,19 +1349,24 @@ void CrossPointWebServer::onWebSocketEvent(uint8_t num, WStype_t type, uint8_t*
     }
 
     case WStype_BIN: {
-      if (!wsUploadInProgress || !wsUploadFile) {
+      if (!wsUploadInProgress || !wsUploadFile || num != wsUploadClientNum) {
         wsServer->sendTXT(num, "ERROR:No upload in progress");
         return;
       }
 
       // Write binary data directly to file
+      size_t remaining = wsUploadSize - wsUploadReceived;
+      if (length > remaining) {
+        abortWsUpload("WS");
+        wsServer->sendTXT(num, "ERROR:Upload overflow");
+        return;
+      }
       esp_task_wdt_reset();
       size_t written = wsUploadFile.write(payload, length);
       esp_task_wdt_reset();
 
       if (written != length) {
-        wsUploadFile.close();
+        abortWsUpload("WS");
-        wsUploadInProgress = false;
         wsServer->sendTXT(num, "ERROR:Write failed - disk full?");
         return;
       }
@@ -1296,17 +1374,17 @@ void CrossPointWebServer::onWebSocketEvent(uint8_t num, WStype_t type, uint8_t*
       wsUploadReceived += written;
 
       // Send progress update (every 64KB or at end)
-      static size_t lastProgressSent = 0;
+      if (wsUploadReceived - wsLastProgressSent >= 65536 || wsUploadReceived >= wsUploadSize) {
-      if (wsUploadReceived - lastProgressSent >= 65536 || wsUploadReceived >= wsUploadSize) {
         String progress = "PROGRESS:" + String(wsUploadReceived) + ":" + String(wsUploadSize);
         wsServer->sendTXT(num, progress);
-        lastProgressSent = wsUploadReceived;
+        wsLastProgressSent = wsUploadReceived;
       }
 
       // Check if upload complete
       if (wsUploadReceived >= wsUploadSize) {
         wsUploadFile.close();
         wsUploadInProgress = false;
+        wsUploadClientNum = 255;
 
         wsLastCompleteName = wsUploadFileName;
         wsLastCompleteSize = wsUploadSize;
@@ -1325,7 +1403,7 @@ void CrossPointWebServer::onWebSocketEvent(uint8_t num, WStype_t type, uint8_t*
         clearEpubCacheIfNeeded(filePath);
 
         wsServer->sendTXT(num, "DONE");
-        lastProgressSent = 0;
+        wsLastProgressSent = 0;
       }
       break;
     }

src/network/CrossPointWebServer.h

@@ -81,6 +81,7 @@ class CrossPointWebServer {
   // WebSocket upload state
   void onWebSocketEvent(uint8_t num, WStype_t type, uint8_t* payload, size_t length);
   static void wsEventCallback(uint8_t num, WStype_t type, uint8_t* payload, size_t length);
+  void abortWsUpload(const char* tag);
 
   // File scanning
   void scanFiles(const char* path, const std::function<void(FileInfo)>& callback) const;
@@ -89,6 +90,7 @@ class CrossPointWebServer {
 
   // Request handlers
   void handleRoot() const;
+  void handleJszip() const;
   void handleNotFound() const;
   void handleStatus() const;
   void handleFileList() const;