crosspoint-reader-mod/mod/docs/upstream-sync.md

Upstream Sync Guide

This document describes how to keep mod/master synchronized with upstream/master (the crosspoint-reader/crosspoint-reader repository) without creating duplicate code or divergence that becomes painful to resolve.

Branch Model

upstream/master  ──────●────●────●────●────●────●──── (upstream development)
                       │                        │
                       │                        │
mod/master       ──────┴──M1──M2──M3───────────┴──M1'──M2'──M3'────
                          ▲                        ▲
                     mod features              mod features
                     on old base               re-applied on new base

• master: Mirror of upstream/master. Updated with git fetch upstream && git merge upstream/master. Never commit mod code here.
• mod/master: The mod branch. Based on upstream/master with mod-exclusive patches applied on top.
• mod/backup-*: Snapshot branches created before each major sync for safety.

The Golden Rule

Never cherry-pick individual upstream PRs into mod/master.

Instead, sync the full upstream/master branch and rebase/replay mod patches on top. Cherry-picking creates shadow copies of upstream commits that:

1. Produce false conflicts when the real PR is later merged upstream
2. Diverge silently if upstream amends the PR after merge (e.g., via a follow-up fix)
3. Make git log --left-right unreliable for tracking what's mod-only vs. upstream
4. Accumulate -- after 50 cherry-picks, you have 50 duplicate commits that must be identified and dropped during every future sync

If an upstream PR is not yet merged and you want its functionality, port the feature (adapted to the mod's codebase) as a mod-exclusive commit with a clear commit message referencing the upstream PR number. Do not copy the upstream commit verbatim. Example:

feat: port upstream word-width cache optimization

Adapted from upstream PR #1027 (not yet merged).
Re-implemented against mod/master's current text layout code.
If/when #1027 is merged upstream, this commit should be dropped
during the next sync and the upstream version used instead.

Sync Procedure

When to Sync

• Weekly during active upstream development periods
• Immediately after a major upstream refactor is merged (e.g., ActivityManager, settings migration)
• Before porting any new upstream PR -- sync first, then port against the latest base

Step-by-Step

1. Prepare

# Ensure clean working tree
git stash  # if needed

# Fetch latest upstream
git fetch upstream

# Update the master mirror
git checkout master
git merge upstream/master  # should always fast-forward
git push origin master

# Create a backup of current mod/master
git branch mod/backup-pre-sync-$(date +%Y-%m-%d) mod/master

2. Identify What Changed

# How many new upstream commits since last sync?
git rev-list --count mod/master..upstream/master

# What are they?
git log --oneline mod/master..upstream/master

# Which mod commits are ahead of upstream?
git log --oneline upstream/master..mod/master

# Divergence summary
git rev-list --left-right --count mod/master...upstream/master
# Output: LEFT RIGHT (LEFT = mod-only commits, RIGHT = new upstream commits)

3. Choose a Sync Strategy

If divergence is small (< 20 new upstream commits, < 5 mod commits ahead):

# Interactive rebase mod/master onto upstream/master
git checkout mod/master
git rebase -i upstream/master
# Resolve conflicts, drop any commits that are now in upstream
git push origin mod/master --force-with-lease

If divergence is moderate (20-50 upstream commits, some conflicts expected):

# Merge upstream into mod/master
git checkout mod/master
git merge upstream/master
# Resolve conflicts, favoring upstream for any cherry-picked PRs
git push origin mod/master

If divergence is severe (50+ upstream commits, major refactors):

Use the "fresh replay" approach. This is the nuclear option -- create a new branch from upstream/master and manually re-apply every mod feature. It produces the cleanest result but requires the most effort.

Fresh Replay Procedure

a. Assessment

Before starting, quantify the problem:

# Divergence stats
git rev-list --left-right --count mod/master...upstream/master

# Conflict preview (count conflict hunks without actually merging)
git merge-tree $(git merge-base mod/master upstream/master) mod/master upstream/master \
  | grep -c "<<<<<<<"

# Files that both sides modified (highest conflict risk)
git merge-tree $(git merge-base mod/master upstream/master) mod/master upstream/master \
  | grep "^changed in both" | wc -l

b. Classify all mod commits

Separate mod-only commits from upstream-ported commits. Each mod commit falls into one of three buckets:

1. Upstream duplicate -- cherry-picked/ported from a PR that is now merged in upstream/master. These are dropped entirely; the upstream version takes precedence.
2. Upstream port (still unmerged) -- ported from a PR that is still open/unmerged upstream. These must be re-applied, adapted to the new upstream code.
3. Mod-exclusive -- original mod features with no upstream equivalent. These must be re-applied.

# List all mod-only commits
git log --oneline upstream/master..mod/master

# Find commits referencing upstream PR numbers
git log --oneline upstream/master..mod/master | grep -iE "(#[0-9]+|port|upstream)"

# For each referenced PR, check if it's now in upstream
git log upstream/master --oneline | grep "#XXXX"

c. Identify upstream architectural changes

Check for major refactors that will affect how mod code must be re-applied. Common high-impact changes include:

• Activity system changes (e.g., ActivityManager migration -- see docs/contributing/ActivityManager.md)
• Settings format changes (e.g., binary to JSON migration)
• Theme system overhauls
• Font rendering pipeline changes
• Library replacements (e.g., image decoder swaps)
• Class/file renames

# See what upstream changed (summary)
git diff --stat $(git merge-base mod/master upstream/master)...upstream/master | tail -5

# Look for large refactors
git log --oneline upstream/master --not mod/master | grep -iE "(refactor|rename|migrate|replace|remove)"

d. Create the replay branch

# Backup current mod/master
git branch mod/backup-pre-sync-$(date +%Y-%m-%d) mod/master

# Create fresh branch from upstream/master
git checkout -b mod/master-resync upstream/master

e. Replay in phases (low-risk to high-risk)

Work through the mod features in order of conflict risk:

Phase 1 -- New files (low risk): Bring over mod-exclusive files that don't exist in upstream. These are purely additive and rarely conflict. Copy them from the backup branch:

# Example: bring over a new mod file
git checkout mod/backup-pre-sync-YYYY-MM-DD -- src/activities/settings/NtpSyncActivity.cpp
git checkout mod/backup-pre-sync-YYYY-MM-DD -- src/activities/settings/NtpSyncActivity.h

New files will need adaptation if they depend on APIs that changed upstream (e.g., Activity base class, settings system). Fix compilation errors as you go.

Phase 2 -- Core file modifications (high risk): Re-apply modifications to files that also changed upstream. Do NOT cherry-pick or copy entire files from the old mod branch -- instead, read the old mod's diff for each file and manually re-apply the mod's intent against the new upstream code. Key files (in typical priority order):

• platformio.ini -- mod build flags
• src/main.cpp -- mod activity registration
• Settings files -- mod settings in the new format
• Activity files modified by mod (HomeActivity, EpubReaderActivity, menus, etc.)
• Renderer and HAL files
• Theme files

# See what the mod changed in a specific file vs the merge-base
git diff $(git merge-base mod/master upstream/master)...mod/backup-pre-sync-YYYY-MM-DD \
  -- src/activities/home/HomeActivity.cpp

# See what upstream changed in that same file
git diff $(git merge-base mod/master upstream/master)...upstream/master \
  -- src/activities/home/HomeActivity.cpp

Phase 3 -- Re-port unmerged upstream PRs: For upstream PRs that were ported into the mod but aren't yet merged upstream, re-apply each one against the new codebase. Check the ported PR tracking table in mod/prs/MERGED.md for context on each port.

f. Verify and finalize

# Build
pio run

# Check for conflict markers
grep -r "<<<<<<" src/ lib/ --include="*.cpp" --include="*.h"

# Run clang-format
./bin/clang-format-fix

# Verify divergence: LEFT should be mod-only, RIGHT should be 0
git rev-list --left-right --count mod/master-resync...upstream/master

# When satisfied, update mod/master
git checkout mod/master
git reset --hard mod/master-resync
git push origin mod/master --force-with-lease

g. Post-sync cleanup

• Update mod/prs/MERGED.md -- remove entries for PRs now in upstream, update status for remaining ports
• Update mod README with new base commit
• Delete the backup branch after confirming everything works on-device

4. Handle Previously-Ported Upstream PRs

During rebase or merge, you will encounter conflicts where the mod cherry-picked an upstream PR that has since been merged natively. Resolution:

• If the upstream PR is now in upstream/master: Drop the mod's cherry-pick commit entirely. The upstream version takes precedence because it may have been amended by follow-up commits.
• If the upstream PR is still unmerged: Keep the mod's port, but verify it still applies cleanly against the updated codebase.

To check which mod commits reference upstream PRs:

# List mod-only commits that reference upstream PR numbers
git log --oneline upstream/master..mod/master | grep -iE "(port|upstream|PR #)"

To check if a specific upstream PR has been merged:

# Check if PR #XXXX is in upstream/master
git log upstream/master --oneline | grep "#XXXX"

5. Verify

# Build
pio run

# Check for leftover conflict markers
grep -r "<<<<<<" src/ lib/ --include="*.cpp" --include="*.h"

# Run clang-format
./bin/clang-format-fix

# Verify the divergence is what you expect
git rev-list --left-right --count mod/master...upstream/master
# LEFT should be only mod-exclusive commits, RIGHT should be 0

6. Document

After a successful sync, update the mod README with the new base upstream commit and note any dropped or reworked ports.

Commit Message Conventions for Mod Commits

Use these prefixes to make mod commits easy to identify and filter during future syncs:

• mod: -- Mod-specific feature or change (e.g., mod: add clock settings tab)
• feat: / fix: / perf: -- Standard prefixes, but for mod-original work
• port: -- Feature ported from an unmerged upstream PR (e.g., port: upstream PR #1027 word-width cache)

Always include in the commit body:

• Which upstream PR it's based on (if any)
• Whether it should be dropped when that PR merges upstream

Tracking Ported PRs

Detailed documentation for each ported PR lives in mod/prs/MERGED.md. That file contains full context on what was changed, how it differs from the upstream PR, and notable discussion.

Additionally, keep the quick-reference status table below up to date during each sync. This table answers the question every sync needs answered: "which ports are still relevant?"

Upstream PR	Description	Upstream Status	Competing/Related PRs	Action on Next Sync
#857	Dictionary word lookup	OPEN	None	Keep until merged upstream
#1003	Image placeholders during decode	OPEN	#1291 (MERGED, user image display setting)	Evaluate -- #1291 may cover this
#1019	File extensions in file browser	OPEN	#1260 (MERGED, MyLibrary->FileBrowser rename)	Keep, adapt to rename
#1027	Word-width cache + hyphenation early exit	OPEN	#1168 (MERGED, fixed-point layout), #873 (MERGED, kerning)	Needs complete rework against new text layout
#1038	std::list to std::vector in text layout	MERGED	--	DROP on next sync (now in upstream)
#1045	Shorten "Forget Wifi" labels	MERGED	--	DROP on next sync (now in upstream)
#1037	Decomposed character hyphenation/rendering	MERGED	--	DROP on next sync (now in upstream)
#1055	Byte-level framebuffer writes	OPEN	#1141 (MERGED, wrapped text in GfxRender)	Keep, adapt to GfxRenderer changes
#1068	URL hyphenation fix	OPEN	None	Keep until merged upstream
#1090	KOReader push progress + sleep	OPEN	#946 (OPEN, sync streamlining)	Evaluate overlap with #946
#1143	TOC fragment navigation	OPEN (draft)	#1172 (OPEN, multi-spine TOC)	Keep until merged upstream
#1172	Multi-spine TOC items	OPEN (draft)	#1143 (OPEN, TOC fragment nav)	Keep until merged upstream
#1185	KOReader document hash cache	OPEN	#1286 (OPEN, OPDS filename matching)	Keep until merged upstream
#1209	Multiple OPDS servers	OPEN	#1214 (OPEN, author folders)	Keep until merged upstream
#1217	KOReader sync improvements	OPEN	#946 (OPEN, sync streamlining)	Evaluate overlap with #946
#1311	Fix inter-word spacing rounding error	OPEN	#1168 (MERGED, fixed-point layout), #873 (MERGED, kerning)	Keep until merged upstream
#1320	JPEG resource cleanup (RAII)	OPEN	None	Keep until merged upstream
#1322	Early exit on fillUncompressedSizes	MERGED	None	DROP on next sync (now in upstream)
#1325	Settings tab label	OPEN	None	Keep until merged upstream
#1329	Reader utils refactor	MERGED	#1143 (dependent)	DROP on next sync (now in upstream)

Last updated: 2026-03-08

How to update this table during a sync

1. For each row, check if the upstream PR has been merged: git log upstream/master --oneline | grep "#XXXX"
2. If merged: change Action to "DROP" and remove the port commit during the sync
3. If still open: check if competing/related PRs have merged that affect the port
4. After the sync: remove dropped rows, add any new ports, update the date

What NOT to Do

1. Don't cherry-pick upstream commits. See golden rule above.
2. Don't let mod/master fall more than ~2 weeks behind upstream/master. The longer you wait, the harder the sync.
3. Don't resolve conflicts by "taking ours" for upstream files without understanding why. The upstream version is almost always correct for code that isn't mod-specific.
4. Don't merge mod/master into master. The master branch is a clean mirror of upstream.
5. Don't port an upstream PR without first syncing. You'll be porting against a stale base, making the next sync harder.
6. Don't create mod feature branches off of master. Always branch from mod/master.