| commit 40221266e98c6cdeb9f08ab2f75a4995519a2407 parent c179031662fc20771c7df0fae552aa337f764c33 Author: sqzer <110329244+sqzer-x@users.noreply.github.com> Date: Sat, 25 Jul 2026 20:57:56 +0900 Reject the Pagefind Korean segmentation patch, with numbers The plan called for a two-line patch to Pagefind — add charabia's "korean" feature, add `| "ko"` to `should_segment` — on the theory that Korean is unsearchable without morphological segmentation. Measured it against a 2000-page Korean corpus with grep-derived ground truth. The theory is wrong twice over. Wrong once: stock Pagefind already has full recall for the ordinary Korean query. Pagefind prefix-matches the last word of a query, and Korean attaches particles as suffixes, so `생성기` is a prefix of 생성기는/를/가. Recall for `템플릿`, `저작권`, `하이라이팅`, `캐시`, `병렬` came back at 1018, 1013, 1019, 1033, 1006 — the ground truth to the page. Wrong twice, and this is the expensive one: the patch destroys that. ko-dic is a general-purpose Korean dictionary and does not know loanwords, so it shreds them — `템플릿은` becomes 템플/릿/은, `하이라이팅도` becomes 하이/라이팅/도. Technical documentation is made of loanwords. The same five queries drop to 27, 28, 31, 917, 967. A 97% loss on the words this project's own docs are written in. The browser cannot rescue it either. Its `needsWordSegmentation` lists the same three languages, and adding "ko" would be a no-op: `Intl.Segmenter("ko")` returns 어절 whole, because ICU ships no Korean dictionary breaker. There is no way to run the same analyzer on both sides, so an index of morphemes can only disagree with a query that isn't one. The `extended` feature goes too. It exists to segment zh/ja/th, and it costs 48.6 MB of jieba and lindera-unidic to do it. Korean search results from a build with it and a build without it were identical, query for query. What stock does not find is a morpheme inside a compound written without spaces — 최적화 inside 검색엔진최적화. Our Korean docs space their terms, so this is nearly empty in practice. If it ever isn't, the answer is not this patch: keep the 어절 as the primary token and add morphemes as compound parts at the same position, which gets the compounds without giving up the prefix match. That was measured too, and it holds. Also noted for the integration: Pagefind 1.5.2 does not declare the tokio features it uses, so `default-features = false` fails to compile until the consumer enables `fs` and `io-util`. Diffstat:
|
| 1 file changed, 8 insertion(+), 1 deletion(-) |
diff --git a/Cargo.toml b/Cargo.toml @@ -60,7 +60,14 @@ walkdir = "2" # --- M3 --- # pagefind 기본 피처 serve가 actix-web + actix-files를 끌고 온다. 반드시 끌 것. -# pagefind = { version = "1.5", default-features = false, features = ["extended"], optional = true } +# 그러면 이번엔 pagefind가 선언을 빠뜨린 tokio 피처를 우리가 채워야 한다: +# 위 tokio에 "fs", "io-util" 추가. 없으면 AsyncWriteExt를 못 찾아 컴파일이 깨진다. +# ⚠️ extended 피처는 켜지 않는다. 켜는 건 zh/ja/th 분절뿐인데(jieba + lindera-unidic) +# 바이너리가 +48.6 MB 늘고, 한국어 검색 결과는 켠 빌드와 끈 빌드가 완전히 동일하다. +# 한국어를 형태소로 쪼개는 charabia "korean"은 더 나쁘다 — ko-dic이 외래어를 모르는 +# 범용 국어사전이라 `템플릿은`을 `템플/릿/은`으로 뭉개서 recall이 무너진다. +# 2000페이지 한국어 코퍼스 실측: 정답 1018건 → 27건. +# pagefind = { version = "1.5", default-features = false, optional = true } # --- M3 이후 --- # rayon = "1" # 병렬 렌더. 문서 사이트 ~60페이지엔 불필요하고, 출력 URL 충돌 시 |