crosspoint-reader-mod/mod/docs/upstream-sync.md

# Upstream Sync Guide

This document describes how to keep `mod/master` synchronized with `upstream/master` (the [crosspoint-reader/crosspoint-reader](https://github.com/crosspoint-reader/crosspoint-reader) repository) without creating duplicate code or divergence that becomes painful to resolve.

## Branch Model

```
upstream/master  ──────●────●────●────●────●────●──── (upstream development)
                       │                        │
                       │                        │
mod/master       ──────┴──M1──M2──M3───────────┴──M1'──M2'──M3'────
                          ▲                        ▲
                     mod features              mod features
                     on old base               re-applied on new base
```

- **`master`**: Mirror of `upstream/master`. Updated with `git fetch upstream && git merge upstream/master`. Never commit mod code here.
- **`mod/master`**: The mod branch. Based on `upstream/master` with mod-exclusive patches applied on top.
- **`mod/backup-*`**: Snapshot branches created before each major sync for safety.

## The Golden Rule

> **Never cherry-pick individual upstream PRs into `mod/master`.**

Instead, sync the full `upstream/master` branch and rebase/replay mod patches on top. Cherry-picking creates shadow copies of upstream commits that:

1. Produce false conflicts when the real PR is later merged upstream
2. Diverge silently if upstream amends the PR after merge (e.g., via a follow-up fix)
3. Make `git log --left-right` unreliable for tracking what's mod-only vs. upstream
4. Accumulate -- after 50 cherry-picks, you have 50 duplicate commits that must be identified and dropped during every future sync

If an upstream PR is not yet merged and you want its functionality, port the *feature* (adapted to the mod's codebase) as a mod-exclusive commit with a clear commit message referencing the upstream PR number. Do not copy the upstream commit verbatim. Example:

```
feat: port upstream word-width cache optimization

Adapted from upstream PR #1027 (not yet merged).
Re-implemented against mod/master's current text layout code.
If/when #1027 is merged upstream, this commit should be dropped
during the next sync and the upstream version used instead.
```

## Sync Procedure

### When to Sync

- **Weekly** during active upstream development periods
- **Immediately** after a major upstream refactor is merged (e.g., ActivityManager, settings migration)
- **Before** porting any new upstream PR -- sync first, then port against the latest base

### Step-by-Step

#### 1. Prepare

```bash
# Ensure clean working tree
git stash  # if needed

# Fetch latest upstream
git fetch upstream

# Update the master mirror
git checkout master
git merge upstream/master  # should always fast-forward
git push origin master

# Create a backup of current mod/master
git branch mod/backup-pre-sync-$(date +%Y-%m-%d) mod/master
```

#### 2. Identify What Changed

```bash
# How many new upstream commits since last sync?
git rev-list --count mod/master..upstream/master

# What are they?
git log --oneline mod/master..upstream/master

# Which mod commits are ahead of upstream?
git log --oneline upstream/master..mod/master

# Divergence summary
git rev-list --left-right --count mod/master...upstream/master
# Output: LEFT RIGHT (LEFT = mod-only commits, RIGHT = new upstream commits)
```

#### 3. Choose a Sync Strategy

**If divergence is small (< 20 new upstream commits, < 5 mod commits ahead):**

```bash
# Interactive rebase mod/master onto upstream/master
git checkout mod/master
git rebase -i upstream/master
# Resolve conflicts, drop any commits that are now in upstream
git push origin mod/master --force-with-lease
```

**If divergence is moderate (20-50 upstream commits, some conflicts expected):**

```bash
# Merge upstream into mod/master
git checkout mod/master
git merge upstream/master
# Resolve conflicts, favoring upstream for any cherry-picked PRs
git push origin mod/master
```

**If divergence is severe (50+ upstream commits, major refactors):**

Use the "fresh replay" approach. This is the nuclear option -- create a new branch from `upstream/master` and manually re-apply every mod feature. It produces the cleanest result but requires the most effort.

##### Fresh Replay Procedure

**a. Assessment**

Before starting, quantify the problem:

```bash
# Divergence stats
git rev-list --left-right --count mod/master...upstream/master

# Conflict preview (count conflict hunks without actually merging)
git merge-tree $(git merge-base mod/master upstream/master) mod/master upstream/master \
  | grep -c "<<<<<<<"

# Files that both sides modified (highest conflict risk)
git merge-tree $(git merge-base mod/master upstream/master) mod/master upstream/master \
  | grep "^changed in both" | wc -l
```

**b. Classify all mod commits**

Separate mod-only commits from upstream-ported commits. Each mod commit falls into one of three buckets:

1. **Upstream duplicate** -- cherry-picked/ported from a PR that is now merged in `upstream/master`. These are dropped entirely; the upstream version takes precedence.
2. **Upstream port (still unmerged)** -- ported from a PR that is still open/unmerged upstream. These must be re-applied, adapted to the new upstream code.
3. **Mod-exclusive** -- original mod features with no upstream equivalent. These must be re-applied.

```bash
# List all mod-only commits
git log --oneline upstream/master..mod/master

# Find commits referencing upstream PR numbers
git log --oneline upstream/master..mod/master | grep -iE "(#[0-9]+|port|upstream)"

# For each referenced PR, check if it's now in upstream
git log upstream/master --oneline | grep "#XXXX"
```

**c. Identify upstream architectural changes**

Check for major refactors that will affect how mod code must be re-applied. Common high-impact changes include:

- Activity system changes (e.g., ActivityManager migration -- see `docs/contributing/ActivityManager.md`)
- Settings format changes (e.g., binary to JSON migration)
- Theme system overhauls
- Font rendering pipeline changes
- Library replacements (e.g., image decoder swaps)
- Class/file renames

```bash
# See what upstream changed (summary)
git diff --stat $(git merge-base mod/master upstream/master)...upstream/master | tail -5

# Look for large refactors
git log --oneline upstream/master --not mod/master | grep -iE "(refactor|rename|migrate|replace|remove)"
```

**d. Create the replay branch**

```bash
# Backup current mod/master
git branch mod/backup-pre-sync-$(date +%Y-%m-%d) mod/master

# Create fresh branch from upstream/master
git checkout -b mod/master-resync upstream/master
```

**e. Replay in phases (low-risk to high-risk)**

Work through the mod features in order of conflict risk:

*Phase 1 -- New files (low risk):* Bring over mod-exclusive files that don't exist in upstream. These are purely additive and rarely conflict. Copy them from the backup branch:

```bash
# Example: bring over a new mod file
git checkout mod/backup-pre-sync-YYYY-MM-DD -- src/activities/settings/NtpSyncActivity.cpp
git checkout mod/backup-pre-sync-YYYY-MM-DD -- src/activities/settings/NtpSyncActivity.h
```

New files will need adaptation if they depend on APIs that changed upstream (e.g., Activity base class, settings system). Fix compilation errors as you go.

*Phase 2 -- Core file modifications (high risk):* Re-apply modifications to files that also changed upstream. Do NOT cherry-pick or copy entire files from the old mod branch -- instead, read the old mod's diff for each file and manually re-apply the mod's *intent* against the new upstream code. Key files (in typical priority order):

- `platformio.ini` -- mod build flags
- `src/main.cpp` -- mod activity registration
- Settings files -- mod settings in the new format
- Activity files modified by mod (HomeActivity, EpubReaderActivity, menus, etc.)
- Renderer and HAL files
- Theme files

```bash
# See what the mod changed in a specific file vs the merge-base
git diff $(git merge-base mod/master upstream/master)...mod/backup-pre-sync-YYYY-MM-DD \
  -- src/activities/home/HomeActivity.cpp

# See what upstream changed in that same file
git diff $(git merge-base mod/master upstream/master)...upstream/master \
  -- src/activities/home/HomeActivity.cpp
```

*Phase 3 -- Re-port unmerged upstream PRs:* For upstream PRs that were ported into the mod but aren't yet merged upstream, re-apply each one against the new codebase. Check the ported PR tracking table in `mod/prs/MERGED.md` for context on each port.

**f. Verify and finalize**

```bash
# Build
pio run

# Check for conflict markers
grep -r "<<<<<<" src/ lib/ --include="*.cpp" --include="*.h"

# Run clang-format
./bin/clang-format-fix

# Verify divergence: LEFT should be mod-only, RIGHT should be 0
git rev-list --left-right --count mod/master-resync...upstream/master

# When satisfied, update mod/master
git checkout mod/master
git reset --hard mod/master-resync
git push origin mod/master --force-with-lease
```

**g. Post-sync cleanup**

- Update `mod/prs/MERGED.md` -- remove entries for PRs now in upstream, update status for remaining ports
- Update mod README with new base commit
- Delete the backup branch after confirming everything works on-device

#### 4. Handle Previously-Ported Upstream PRs

During rebase or merge, you will encounter conflicts where the mod cherry-picked an upstream PR that has since been merged natively. Resolution:

- **If the upstream PR is now in `upstream/master`**: Drop the mod's cherry-pick commit entirely. The upstream version takes precedence because it may have been amended by follow-up commits.
- **If the upstream PR is still unmerged**: Keep the mod's port, but verify it still applies cleanly against the updated codebase.

To check which mod commits reference upstream PRs:

```bash
# List mod-only commits that reference upstream PR numbers
git log --oneline upstream/master..mod/master | grep -iE "(port|upstream|PR #)"
```

To check if a specific upstream PR has been merged:

```bash
# Check if PR #XXXX is in upstream/master
git log upstream/master --oneline | grep "#XXXX"
```

#### 5. Verify

```bash
# Build
pio run

# Check for leftover conflict markers
grep -r "<<<<<<" src/ lib/ --include="*.cpp" --include="*.h"

# Run clang-format
./bin/clang-format-fix

# Verify the divergence is what you expect
git rev-list --left-right --count mod/master...upstream/master
# LEFT should be only mod-exclusive commits, RIGHT should be 0
```

#### 6. Document

After a successful sync, update the mod README with the new base upstream commit and note any dropped or reworked ports.

## Commit Message Conventions for Mod Commits

Use these prefixes to make mod commits easy to identify and filter during future syncs:

- `mod:` -- Mod-specific feature or change (e.g., `mod: add clock settings tab`)
- `feat:` / `fix:` / `perf:` -- Standard prefixes, but for mod-original work
- `port:` -- Feature ported from an unmerged upstream PR (e.g., `port: upstream PR #1027 word-width cache`)

Always include in the commit body:
- Which upstream PR it's based on (if any)
- Whether it should be dropped when that PR merges upstream

## Tracking Ported PRs

Detailed documentation for each ported PR lives in [`mod/prs/MERGED.md`](../prs/MERGED.md). That file contains full context on what was changed, how it differs from the upstream PR, and notable discussion.

Additionally, keep the quick-reference status table below up to date during each sync. This table answers the question every sync needs answered: "which ports are still relevant?"

| Upstream PR | Description | Upstream Status | Competing/Related PRs | Action on Next Sync |
|---|---|---|---|---|
| #857 | Dictionary word lookup | OPEN | None | Keep until merged upstream |
| #1003 | Image placeholders during decode | OPEN | #1291 (MERGED, user image display setting) | Evaluate -- #1291 may cover this |
| #1019 | File extensions in file browser | OPEN | #1260 (MERGED, MyLibrary->FileBrowser rename) | Keep, adapt to rename |
| #1027 | Word-width cache + hyphenation early exit | OPEN | #1168 (MERGED, fixed-point layout), #873 (MERGED, kerning) | Needs complete rework against new text layout |
| #1038 | std::list to std::vector in text layout | MERGED | -- | DROP on next sync (now in upstream) |
| #1045 | Shorten "Forget Wifi" labels | MERGED | -- | DROP on next sync (now in upstream) |
| #1037 | Decomposed character hyphenation/rendering | MERGED | -- | DROP on next sync (now in upstream) |
| #1055 | Byte-level framebuffer writes | OPEN | #1141 (MERGED, wrapped text in GfxRender) | Keep, adapt to GfxRenderer changes |
| #1068 | URL hyphenation fix | OPEN | None | Keep until merged upstream |
| #1090 | KOReader push progress + sleep | OPEN | #946 (OPEN, sync streamlining) | Evaluate overlap with #946 |
| #1185 | KOReader document hash cache | OPEN | #1286 (OPEN, OPDS filename matching) | Keep until merged upstream |
| #1209 | Multiple OPDS servers | OPEN | #1214 (OPEN, author folders) | Keep until merged upstream |
| #1217 | KOReader sync improvements | OPEN | #946 (OPEN, sync streamlining) | Evaluate overlap with #946 |

*Last updated: 2026-03-07*

### How to update this table during a sync

1. For each row, check if the upstream PR has been merged: `git log upstream/master --oneline | grep "#XXXX"`
2. If merged: change Action to "DROP" and remove the port commit during the sync
3. If still open: check if competing/related PRs have merged that affect the port
4. After the sync: remove dropped rows, add any new ports, update the date

## What NOT to Do

1. **Don't cherry-pick upstream commits.** See golden rule above.
2. **Don't let mod/master fall more than ~2 weeks behind upstream/master.** The longer you wait, the harder the sync.
3. **Don't resolve conflicts by "taking ours" for upstream files without understanding why.** The upstream version is almost always correct for code that isn't mod-specific.
4. **Don't merge `mod/master` into `master`.** The `master` branch is a clean mirror of upstream.
5. **Don't port an upstream PR without first syncing.** You'll be porting against a stale base, making the next sync harder.
6. **Don't create mod feature branches off of `master`.** Always branch from `mod/master`.