cottongin/crosspoint-reader-mod
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

perf: Reduce overall flash usage by 30.7% by compressing built-in fonts (#831)

Browse Source 
## Summary

**What is the goal of this PR?**

Compress reader font bitmaps to reduce flash usage by 30.7%.

**What changes are included?**

- New `EpdFontGroup` struct and extended `EpdFontData` with
`groups`/`groupCount` fields
- `--compress` flag in `fontconvert.py`: groups glyphs (ASCII base group
+ groups of 8) and compresses each with raw DEFLATE
- `FontDecompressor` class with 4-slot LRU cache for on-demand
decompression during rendering
- `GfxRenderer` transparently routes bitmap access through
`getGlyphBitmap()` (compressed or direct flash)
- Uses `uzlib` for decompression with minimal heap overhead.
- 48 reader fonts (Bookerly, NotoSans 12-18pt, OpenDyslexic) regenerated
with compression; 5 UI fonts unchanged
- Round-trip verification script (`verify_compression.py`) runs as part
of font generation
## Additional Context

## Flash & RAM

| | baseline | font-compression | Difference |
|--|--------|-----------------|------------|
| Flash (ELF) | 6,302,476 B (96.2%) | 4,365,022 B (66.6%) | -1,937,454 B
(-30.7%) |
| firmware.bin | 6,468,192 B | 4,531,008 B | -1,937,184 B (-29.9%) |
| RAM | 101,700 B (31.0%) | 103,076 B (31.5%) | +1,376 B (+0.5%) |

## Script-Based Grouping (Cold Cache)

Comparison of uncompressed baseline vs script-based group compression
(4-slot LRU cache, cleared each page). Glyphs are grouped by Unicode
block (ASCII, Latin-1, Latin Extended-A, Combining Marks, Cyrillic,
General Punctuation, etc.) instead of sequential groups of 8.

### Render Time

| | Baseline | Compressed (cold cache) | Difference |
|---|---|---|---|
| **Median** | 414.9 ms | 431.6 ms | +16.7 ms (+4.0%) |
| **Pages** | 37 | 37 | |

### Memory Usage

| | Baseline | Compressed (cold cache) | Difference |
|---|---|---|---|
| **Heap free (median)** | 187.0 KB | 176.3 KB | -10.7 KB |
| **Heap free (min)** | 186.0 KB | 166.5 KB | -19.5 KB |
| **Largest block (median)** | 148.0 KB | 128.0 KB | -20.0 KB |
| **Largest block (min)** | 148.0 KB | 120.0 KB | -28.0 KB |

### Cache Effectiveness

| | Misses/page | Hit rate |
|---|---|---|
| **Compressed (cold cache)** | 2.1 | 99.85% |

------

### AI Usage

While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.

  Did you use AI tools to help write this code? _**YES**_
Implementation was done by Claude Code (Opus 4.6) based on a plan
developed collaboratively. All generated font headers were verified with
an automated round-trip decompression test. The firmware was compiled
successfully but has not yet been tested on-device.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Adrian Wilkins-Caruana
2026-02-19 20:30:15 +11:00
committed by GitHub
parent f16c0e52fd
commit 47aa0dda76
73 changed files with 126417 additions and 246010 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
lib/EpdFont/scripts/convert-builtin-fonts.sh
View File

@@ -14,7 +14,7 @@ for size in ${BOOKERLY_FONT_SIZES[@]}; do
font_name="bookerly_${size}_$(echo $style | tr '[:upper:]' '[:lower:]')"
font_path="../builtinFonts/source/Bookerly/Bookerly-${style}.ttf"
output_path="../builtinFonts/${font_name}.h"
python fontconvert.py $font_name $size $font_path --2bit > $output_path
python fontconvert.py $font_name $size $font_path --2bit --compress > $output_path
echo "Generated $output_path"
done
done
@@ -24,7 +24,7 @@ for size in ${NOTOSANS_FONT_SIZES[@]}; do
font_name="notosans_${size}_$(echo $style | tr '[:upper:]' '[:lower:]')"
font_path="../builtinFonts/source/NotoSans/NotoSans-${style}.ttf"
output_path="../builtinFonts/${font_name}.h"
python fontconvert.py $font_name $size $font_path --2bit > $output_path
python fontconvert.py $font_name $size $font_path --2bit --compress > $output_path
echo "Generated $output_path"
done
done
@@ -34,7 +34,7 @@ for size in ${OPENDYSLEXIC_FONT_SIZES[@]}; do
font_name="opendyslexic_${size}_$(echo $style | tr '[:upper:]' '[:lower:]')"
font_path="../builtinFonts/source/OpenDyslexic/OpenDyslexic-${style}.otf"
output_path="../builtinFonts/${font_name}.h"
python fontconvert.py $font_name $size $font_path --2bit > $output_path
python fontconvert.py $font_name $size $font_path --2bit --compress > $output_path
echo "Generated $output_path"
done
done
@@ -53,3 +53,7 @@ for size in ${UI_FONT_SIZES[@]}; do
done
python fontconvert.py notosans_8_regular 8 ../builtinFonts/source/NotoSans/NotoSans-Regular.ttf > ../builtinFonts/notosans_8_regular.h
echo ""
echo "Running compression verification..."
python verify_compression.py ../builtinFonts/

119
lib/EpdFont/scripts/fontconvert.py
View File

@@ -15,6 +15,7 @@ parser.add_argument("size", type=int, help="font size to use.")
parser.add_argument("fontstack", action="store", nargs='+', help="list of font files, ordered by descending priority.")
parser.add_argument("--2bit", dest="is2Bit", action="store_true", help="generate 2-bit greyscale bitmap instead of 1-bit black and white.")
parser.add_argument("--additional-intervals", dest="additional_intervals", action="append", help="Additional code point intervals to export as min,max. This argument can be repeated.")
parser.add_argument("--compress", dest="compress", action="store_true", help="Compress glyph bitmaps using DEFLATE with group-based compression.")
args = parser.parse_args()
GlyphProps = namedtuple("GlyphProps", ["width", "height", "advance_x", "left", "top", "data_length", "data_offset", "code_point"])
@@ -270,21 +271,115 @@ for index, glyph in enumerate(all_glyphs):
glyph_data.extend([b for b in packed])
glyph_props.append(props)
compress = args.compress
# Build groups for compression
if compress:
# Script-based grouping: glyphs that co-occur in typical text rendering
# are grouped together for efficient LRU caching on the embedded target.
# Since glyphs are in codepoint order, glyphs in the same Unicode block
# are contiguous in the array and form natural groups.
SCRIPT_GROUP_RANGES = [
(0x0000, 0x007F), # ASCII
(0x0080, 0x00FF), # Latin-1 Supplement
(0x0100, 0x017F), # Latin Extended-A
(0x0300, 0x036F), # Combining Diacritical Marks
(0x0400, 0x04FF), # Cyrillic
(0x2000, 0x206F), # General Punctuation
(0x2070, 0x209F), # Superscripts & Subscripts
(0x20A0, 0x20CF), # Currency Symbols
(0x2190, 0x21FF), # Arrows
(0x2200, 0x22FF), # Math Operators
(0xFFFD, 0xFFFD), # Replacement Character
]
def get_script_group(code_point):
for i, (start, end) in enumerate(SCRIPT_GROUP_RANGES):
if start <= code_point <= end:
return i
return -1
groups = [] # list of (first_glyph_index, glyph_count)
current_group_id = None
group_start = 0
group_count = 0
for i, (props, packed) in enumerate(all_glyphs):
sg = get_script_group(props.code_point)
if sg != current_group_id:
if group_count > 0:
groups.append((group_start, group_count))
current_group_id = sg
group_start = i
group_count = 1
else:
group_count += 1
if group_count > 0:
groups.append((group_start, group_count))
# Compress each group
compressed_groups = [] # list of (compressed_bytes, uncompressed_size, glyph_count, first_glyph_index)
compressed_bitmap_data = []
compressed_offset = 0
# Also build modified glyph props with within-group offsets
modified_glyph_props = list(glyph_props)
for first_idx, count in groups:
# Concatenate bitmap data for this group
group_data = b''
for gi in range(first_idx, first_idx + count):
props, packed = all_glyphs[gi]
# Update glyph's dataOffset to be within-group offset
within_group_offset = len(group_data)
old_props = modified_glyph_props[gi]
modified_glyph_props[gi] = GlyphProps(
width=old_props.width,
height=old_props.height,
advance_x=old_props.advance_x,
left=old_props.left,
top=old_props.top,
data_length=old_props.data_length,
data_offset=within_group_offset,
code_point=old_props.code_point,
)
group_data += packed
# Compress with raw DEFLATE (no zlib/gzip header)
compressor = zlib.compressobj(level=9, wbits=-15)
compressed = compressor.compress(group_data) + compressor.flush()
compressed_groups.append((compressed, len(group_data), count, first_idx))
compressed_bitmap_data.extend(compressed)
compressed_offset += len(compressed)
glyph_props = modified_glyph_props
total_compressed = len(compressed_bitmap_data)
total_uncompressed = len(glyph_data)
print(f"// Compression: {total_uncompressed} -> {total_compressed} bytes ({100*total_compressed/total_uncompressed:.1f}%), {len(groups)} groups", file=sys.stderr)
print(f"""/**
* generated by fontconvert.py
* name: {font_name}
* size: {size}
* mode: {'2-bit' if is2Bit else '1-bit'}
* mode: {'2-bit' if is2Bit else '1-bit'}{' compressed: true' if compress else ''}
* Command used: {' '.join(sys.argv)}
*/
#pragma once
#include "EpdFontData.h"
""")
print(f"static const uint8_t {font_name}Bitmaps[{len(glyph_data)}] = {{")
for c in chunks(glyph_data, 16):
print (" " + " ".join(f"0x{b:02X}," for b in c))
print ("};\n");
if compress:
print(f"static const uint8_t {font_name}Bitmaps[{len(compressed_bitmap_data)}] = {{")
for c in chunks(compressed_bitmap_data, 16):
print (" " + " ".join(f"0x{b:02X}," for b in c))
print ("};\n");
else:
print(f"static const uint8_t {font_name}Bitmaps[{len(glyph_data)}] = {{")
for c in chunks(glyph_data, 16):
print (" " + " ".join(f"0x{b:02X}," for b in c))
print ("};\n");
print(f"static const EpdGlyph {font_name}Glyphs[] = {{")
for i, g in enumerate(glyph_props):
@@ -298,6 +393,14 @@ for i_start, i_end in intervals:
offset += i_end - i_start + 1
print ("};\n");
if compress:
print(f"static const EpdFontGroup {font_name}Groups[] = {{")
compressed_offset = 0
for compressed, uncompressed_size, count, first_idx in compressed_groups:
print(f" {{ {compressed_offset}, {len(compressed)}, {uncompressed_size}, {count}, {first_idx} }},")
compressed_offset += len(compressed)
print("};\n")
print(f"static const EpdFontData {font_name} = {{")
print(f" {font_name}Bitmaps,")
print(f" {font_name}Glyphs,")
@@ -307,4 +410,10 @@ print(f" {norm_ceil(face.size.height)},")
print(f" {norm_ceil(face.size.ascender)},")
print(f" {norm_floor(face.size.descender)},")
print(f" {'true' if is2Bit else 'false'},")
if compress:
print(f" {font_name}Groups,")
print(f" {len(compressed_groups)},")
else:
print(f" nullptr,")
print(f" 0,")
print("};")

164
lib/EpdFont/scripts/verify_compression.py Normal file
View File

@@ -0,0 +1,164 @@
#!/usr/bin/env python3
"""
Round-trip verification for compressed font headers.
Parses each generated .h file in the given directory, identifies compressed fonts
(those with a Groups array), decompresses each group, and verifies that
decompression succeeds and all glyph offsets/lengths fall within bounds.
"""
import os
import re
import sys
import zlib
def parse_hex_array(text):
"""Extract bytes from a C hex array string like '{ 0xAB, 0xCD, ... }'"""
hex_vals = re.findall(r'0x([0-9A-Fa-f]{2})', text)
return bytes(int(h, 16) for h in hex_vals)
def parse_groups(text):
"""Parse EpdFontGroup array entries: { compressedOffset, compressedSize, uncompressedSize, glyphCount, firstGlyphIndex }"""
groups = []
for match in re.finditer(r'\{\s*(\d+)\s*,\s*(\d+)\s*,\s*(\d+)\s*,\s*(\d+)\s*,\s*(\d+)\s*\}', text):
groups.append({
'compressedOffset': int(match.group(1)),
'compressedSize': int(match.group(2)),
'uncompressedSize': int(match.group(3)),
'glyphCount': int(match.group(4)),
'firstGlyphIndex': int(match.group(5)),
})
return groups
def parse_glyphs(text):
"""Parse EpdGlyph array entries: { width, height, advanceX, left, top, dataLength, dataOffset }"""
glyphs = []
for match in re.finditer(r'\{\s*(-?\d+)\s*,\s*(-?\d+)\s*,\s*(-?\d+)\s*,\s*(-?\d+)\s*,\s*(-?\d+)\s*,\s*(-?\d+)\s*,\s*(-?\d+)\s*\}', text):
glyphs.append({
'width': int(match.group(1)),
'height': int(match.group(2)),
'advanceX': int(match.group(3)),
'left': int(match.group(4)),
'top': int(match.group(5)),
'dataLength': int(match.group(6)),
'dataOffset': int(match.group(7)),
})
return glyphs
def verify_font_file(filepath):
"""Verify a single font header file. Returns (font_name, success, message)."""
with open(filepath, 'r') as f:
content = f.read()
# Check if this is a compressed font (has Groups array)
groups_match = re.search(r'static const EpdFontGroup (\w+)Groups\[\]', content)
if not groups_match:
return (os.path.basename(filepath), None, "uncompressed, skipping")
font_name = groups_match.group(1)
# Extract bitmap data
bitmap_match = re.search(
r'static const uint8_t ' + re.escape(font_name) + r'Bitmaps\[\d+\]\s*=\s*\{([^}]+)\}',
content, re.DOTALL
)
if not bitmap_match:
return (font_name, False, "could not find Bitmaps array")
compressed_data = parse_hex_array(bitmap_match.group(1))
# Extract groups
groups_array_match = re.search(
r'static const EpdFontGroup ' + re.escape(font_name) + r'Groups\[\]\s*=\s*\{(.+?)\};',
content, re.DOTALL
)
if not groups_array_match:
return (font_name, False, "could not find Groups array")
groups = parse_groups(groups_array_match.group(1))
if not groups:
return (font_name, False, "Groups array parsed to 0 entries; check format")
# Extract glyphs
glyphs_match = re.search(
r'static const EpdGlyph ' + re.escape(font_name) + r'Glyphs\[\]\s*=\s*\{(.+?)\};',
content, re.DOTALL
)
if not glyphs_match:
return (font_name, False, "could not find Glyphs array")
glyphs = parse_glyphs(glyphs_match.group(1))
# Verify each group
for gi, group in enumerate(groups):
# Extract compressed chunk
chunk = compressed_data[group['compressedOffset']:group['compressedOffset'] + group['compressedSize']]
if len(chunk) != group['compressedSize']:
return (font_name, False, f"group {gi}: compressed data truncated (expected {group['compressedSize']}, got {len(chunk)})")
# Decompress with raw DEFLATE
try:
decompressed = zlib.decompress(chunk, -15)
except zlib.error as e:
return (font_name, False, f"group {gi}: decompression failed: {e}")
if len(decompressed) != group['uncompressedSize']:
return (font_name, False, f"group {gi}: size mismatch (expected {group['uncompressedSize']}, got {len(decompressed)})")
# Verify each glyph's data within the group
first = group['firstGlyphIndex']
for j in range(group['glyphCount']):
glyph_idx = first + j
if glyph_idx >= len(glyphs):
return (font_name, False, f"group {gi}: glyph index {glyph_idx} out of range")
glyph = glyphs[glyph_idx]
offset = glyph['dataOffset']
length = glyph['dataLength']
if offset + length > len(decompressed):
return (font_name, False, f"group {gi}, glyph {glyph_idx}: data extends beyond decompressed buffer "
f"(offset={offset}, length={length}, decompressed_size={len(decompressed)})")
return (font_name, True, f"{len(groups)} groups, {len(glyphs)} glyphs OK")
def main():
if len(sys.argv) < 2:
print(f"Usage: {sys.argv[0]} <font_headers_directory>", file=sys.stderr)
sys.exit(1)
font_dir = sys.argv[1]
if not os.path.isdir(font_dir):
print(f"Error: {font_dir} is not a directory", file=sys.stderr)
sys.exit(1)
files = sorted(f for f in os.listdir(font_dir) if f.endswith('.h') and f != 'all.h')
passed = 0
failed = 0
skipped = 0
for filename in files:
filepath = os.path.join(font_dir, filename)
_font_name, success, message = verify_font_file(filepath)
if success is None:
skipped += 1
elif success:
passed += 1
print(f" PASS: {filename} ({message})")
else:
failed += 1
print(f" FAIL: {filename} - {message}")
print(f"\nResults: {passed} passed, {failed} failed, {skipped} skipped (uncompressed)")
if failed > 0:
sys.exit(1)
if __name__ == '__main__':
main()