scripts/subset-fonts.py (10750B)
1 #!/usr/bin/env python3
2 """Rebuild the shipped web fonts from the variable masters in fonts-src/.
3
4 fonts-src/ is deliberately outside public/: Astro copies public/ verbatim, so masters
5 kept there would be published alongside the subsets - 294,772B of masters in dist/fonts/
6 for a site that references two files totalling 140,436B.
7
8 Maintenance tool, not a build step: the outputs are committed to public/fonts/ and the
9 GitHub Actions build never runs this. Re-run it when the corpus gains characters the
10 subsets do not cover. It rewrites every live reference to the old file names (see
11 rewrite(); the measured-results table below keeps the names it was measured with); then
12 run `npm run diagrams`, whose cache is keyed to those names, and commit the new fonts,
13 the rewritten files and src/lib/diagrams/ together. Needs Python 3.10 or later.
14
15 python -m pip install "fonttools[woff]" brotli
16 python scripts/subset-fonts.py
17
18 Both faces are under the SIL Open Font License 1.1, which requires the licence to travel
19 with every copy: OFL-Lora.txt and OFL-Pretendard.txt sit beside the masters in fonts-src/
20 and beside the subsets in public/fonts/, and this script leaves them alone.
21
22 Two faces, two scripts. Lora is the only Latin face the site ships - body, headings,
23 links, nav and the wordmark all resolve to it; code uses the reader's system monospace -
24 and Pretendard covers Hangul. Lora has no Hangul and Pretendard's Latin is switched off
25 by unicode-range, so the two never compete for a codepoint.
26
27 Why each flag is here, measured on this repo's masters by removing one flag at a time
28 from the recipe below and re-encoding in a single session, so the pairs below compare
29 like with like:
30
31 unicodes The dominant saving on Lora, and the only large one. The master carries 778
32 codepoints - 318 Latin Extended and Vietnamese, 242 Cyrillic, 20 combining
33 marks, 6 Greek - against the 105 kept in LATIN below. Cutting to
34 LATIN takes 71,504 -> 22,288. On Pretendard the same idea drops the Latin the
35 unicode-range gate had already made unreachable: 130,528 -> 118,212.
36
37 instancer Worth 49,552B on Pretendard, whose wght axis runs 45-930 against the 400-700
38 the design uses, and whose gvar is 226,842B of the 466,492B sfnt. On Lora it
39 is worth nothing: the axis already *is* 400-700, so there is no range to trim
40 and the two encodings differ by 4B, which is inside woff2's own noise. The
41 call stays because it is what pins the design's range in one readable place,
42 not because it pays for itself on the Latin face.
43
44 features kern + liga, nothing else. GPOS also carries mark attachment and script
45 shaping this site never invokes; dropping the rest takes Lora 23,268 ->
46 22,288. On Pretendard it is worth 28B, i.e. nothing: once the Latin is
47 gone, Hangul has no other features left to drop. Kept for both so the recipe
48 is one recipe.
49
50 liga is kept for Lora's f, whose prominent hook runs into the dot of a
51 following i, and the built corpus renders many `fi`, `fl` and `ffl`
52 sequences. Lora's liga covers f+i and f+l and costs 396B over kern alone.
53 Dropping it would save a third of a kilobyte by breaking every one of them.
54
55 Measured results (bytes), as shipped:
56 lora-var.woff2 84,764 -> latin-9bba8f25.woff2 22,300 (-74%)
57 pretendard-var.woff2 210,008 -> ko-02f8cebd.woff2 118,136 (-44%)
58 english route 22,300
59 korean route 140,436
60
61 woff2 encoding is not byte-deterministic. Re-running this on unchanged masters moves the
62 output by a hundred bytes or so - Pretendard has come out at 118,136, 118,212 and 118,276
63 from identical input - which is why every filename is content-hashed and why the figures
64 above are quoted to compare against each other, never as a checksum.
65
66 Not shipped, deliberately:
67
68 Lora-Italic (wght 400-700, same recipe) ~23,400 0 renders in the whole corpus.
69 Emphasis that satteri-figure.mjs lifts out as a figure caption is upright by
70 design, and every <em> left in dist/ is on a document that resolves to
71 lang="ko", where [slug].astro sets
72 `.post[lang='ko'] .prose em { font-style: normal; font-weight: 500 }` because
73 Pretendard has no italic. The same file's `.prose em { font-style: italic }` is
74 therefore overridden on every <em>, and dist/ holds no <i> or <cite>, so no page requests
75 an italic and none is synthesised - this is 23KB that would never be fetched.
76 If an English post ever uses emphasis, put a Lora-Italic master in fonts-src/,
77 add one JOBS row here and an @font-face with font-style: italic in fonts.css;
78 the recipe is unchanged.
79
80 Three pinned weights instead of one axis 11,872 + 12,396 + 11,912 = 36,180, against
81 22,288 for the axis. The design reaches 400 (body, links, nav, wordmark), 500
82 (headings, Korean emphasis) and 700 (kickers, table headers, the byline's name,
83 strong), and Lora's axis is exactly 400-700, so all three are real instances.
84 One variable file carrying all three is 13,892B cheaper than three static ones
85 and is one request instead of three.
86 """
87 import hashlib, os, pathlib, sys
88 from fontTools.ttLib import TTFont
89 from fontTools.varLib import instancer
90 from fontTools import subset
91
92 ROOT = pathlib.Path(__file__).resolve().parent.parent
93 SRC = ROOT / "fonts-src" # variable masters, never published
94 OUT = ROOT / "public" / "fonts" # the two faces the site serves, beside their licences
95 TMP = ROOT / ".fonts-work"
96
97 # The five Hangul blocks, identical to the unicode-range in fonts.css.
98 KO = "U+1100-11FF,U+3130-318F,U+A960-A97F,U+AC00-D7A3,U+D7B0-D7FF"
99
100 # Printable ASCII plus ten non-ASCII marks. Every .html in dist/ plus every .md in
101 # content/ was scanned as source text, and the non-Hangul characters Lora has come to
102 # exactly these 105 codepoints. A character written as an
103 # entity is not seen that way: the footer's `©` (Base.astro) paints U+00A9, which
104 # Lora has and this set leaves out, so the sign is drawn from the fallback face.
105 #
106 # 27 further non-Hangul codepoints appear in the corpus and are in no shipped face:
107 # arrows, box drawing, circled digits, kana, a Han character, symbols and emoji. They
108 # fall to a system font. Adding them is not an option - Lora has none of them.
109 LATIN = "U+0020-007E,U+00B7,U+2013-2014,U+2018-2019,U+201C-201D,U+2022,U+2026,U+2248"
110
111 JOBS = [
112 # master, stem, axis limits, unicodes
113 ("lora-var.woff2", "latin", {"wght": (400, 700)}, LATIN),
114 ("pretendard-var.woff2", "ko", {"wght": (400, 700)}, KO),
115 ]
116
117
118 def build(master, stem, limits, unicodes):
119 src = SRC / master
120 if not src.exists():
121 sys.exit(f"missing master: {src}")
122 font = TTFont(src)
123 font = instancer.instantiateVariableFont(
124 font, limits, updateFontNames=False, optimize=True
125 )
126 TMP.mkdir(exist_ok=True)
127 flat = TMP / f"_{stem}.ttf"
128 font.flavor = None
129 font.save(flat)
130
131 out = TMP / f"{stem}.woff2"
132 subset.main([
133 str(flat),
134 f"--output-file={out}",
135 "--flavor=woff2",
136 "--no-hinting",
137 "--desubroutinize",
138 "--layout-features=kern,liga",
139 "--drop-tables+=DSIG",
140 f"--unicodes={unicodes}",
141 ])
142 flat.unlink()
143
144 digest = hashlib.sha256(out.read_bytes()).hexdigest()[:8]
145 # public/ is copied verbatim by Vite, so nothing hashes these filenames for us.
146 # GitHub Pages pins Cache-Control at max-age=600 and cannot be configured, which
147 # makes the filename the only cache-invalidation handle we have.
148 final = OUT / f"{stem}-{digest}.woff2"
149 OUT.mkdir(parents=True, exist_ok=True)
150 for stale in OUT.glob(f"{stem}-*.woff2"):
151 if stale != final:
152 stale.unlink()
153 final.write_bytes(out.read_bytes())
154 out.unlink()
155 return final.name, final.stat().st_size
156
157
158 def rewrite(names):
159 """Point fonts.css, Base.astro, tokens.css and diagrams.mjs at the files just written.
160
161 woff2 encoding is not byte-deterministic, so the digest moves on every run even when
162 the input does not. Hand-syncing filenames across four files after each rebuild is a
163 guaranteed source of a silent 404 and a page rendered entirely in fallback, so the
164 script owns the references instead of documenting them.
165 """
166 import re
167 targets = [
168 ROOT / "src" / "styles" / "fonts.css",
169 ROOT / "src" / "layouts" / "Base.astro",
170 # tokens.css cites the shipped Latin file by its hashed name in the type-scale
171 # derivation comment, so it goes stale on every rebuild exactly like the other
172 # files. It carries no url(), only prose, so a stale name here is a misleading
173 # citation rather than a 404 - but it is the same hand-sync this function exists
174 # to remove.
175 ROOT / "src" / "styles" / "tokens.css",
176 # diagrams.mjs names both files in DIAGRAM_FONTS (as public/fonts/..., which the
177 # pattern below matches), and that list is both what render-diagrams.mjs loads
178 # into the browser and part of every diagram's cache key. Left stale, the keys do
179 # not move, so the committed SVGs stay keyed to fonts that no longer exist, and
180 # the next diagram render stops on a missing font file.
181 ROOT / "src" / "lib" / "diagrams.mjs",
182 ]
183 for t in targets:
184 text = t.read_text(encoding="utf-8")
185 for stem, name in names.items():
186 text = re.sub(rf"/fonts/{stem}-[0-9a-f]{{8}}\.woff2", f"/fonts/{name}", text)
187 # LF, as .gitattributes keeps the working tree; the default writes CRLF on Windows.
188 t.write_text(text, encoding="utf-8", newline="\n")
189 print(f" updated {t.relative_to(ROOT)}")
190
191
192 def main():
193 print(f"{'file':<34}{'bytes':>9}")
194 names, sizes = {}, {}
195 for master, stem, limits, unicodes in JOBS:
196 name, size = build(master, stem, limits, unicodes)
197 names[stem], sizes[stem] = name, size
198 print(f"{name:<34}{size:>9,}")
199 print("-" * 43)
200 print(f"{'latin only (english route)':<34}{sizes['latin']:>9,}")
201 print(f"{'+ korean face (korean route)':<34}{sizes['latin'] + sizes['ko']:>9,}")
202 print()
203 rewrite(names)
204 print()
205 print("Next: run `npm run diagrams`. The diagram cache is keyed to these file names,")
206 print("so every diagram re-renders; commit src/lib/diagrams/ with the fonts and the")
207 print("files updated above.")
208
209
210 if __name__ == "__main__":
211 main()