docs/content/features/search.md (4259B)
1 +++
2 title = "Search"
3 description = "A substring index, one file per language, and why it is not a word index"
4 weight = 20
5 toc = true
6 +++
7
8 Every build writes one index per language:
9
10 ```
11 public/
12 ├── search-en.json
13 └── search-ko.json
14 ```
15
16 Each row is a page: its title, description, section, URL and body text. The
17 client fetches the file for the current language the first time someone opens
18 search, and scans it for substrings.
19
20 ## Why substrings
21
22 The usual approach is a word index: split the text into words at build time,
23 look the query's words up at search time. It is smaller and it is faster, and
24 for Korean it is wrong.
25
26 Korean attaches particles to nouns, so `생성기` appears in running text as
27 `생성기는`, `생성기를`, `생성기가`. A word index can survive that with prefix
28 matching. What it cannot survive is that Korean writes compounds without
29 spaces: `최적화` inside `검색엔진최적화` is not a prefix of anything, and a word
30 index will never return it. Neither will it find `존성` inside `의존성이`.
31
32 The apparent fix is to run a morphological analyser at index time, and it makes
33 things worse. The dictionaries are general-purpose and do not know loanwords,
34 which is most of a technical vocabulary: `템플릿은` comes back as
35 `템플`+`릿`+`은`, and the word `템플릿` stops matching the pages that are about
36 it. Measured on a 2000-page Korean corpus, recall for `템플릿` fell from 1018
37 pages to 27.
38
39 Nor can the browser be taught to compensate. `Intl.Segmenter("ko")` returns
40 Korean 어절 whole — ICU ships dictionary-based breakers for Chinese, Japanese
41 and Thai, and none for Korean — so an index of morphemes has no way to agree
42 with the query typed against it.
43
44 Scanning the text for substrings has none of these problems, in any language,
45 and it costs a JSON file.
46
47 ## What it costs
48
49 The index is the body text of every page. For this site that is around 80 KB
50 per language, fetched once, on the first search. It grows linearly
51 with your documentation, and there is a size past which this is the wrong
52 design — but that size is far beyond a documentation site, and reaching it is a
53 better problem than shipping search that cannot find your own words.
54
55 A site that ships no search UI should not pay even that:
56
57 ```toml
58 [search]
59 enabled = false
60 ```
61
62 skips the index entirely.
63
64 Templates see the same switch as `site.search` — the current language's index
65 URL, or nothing:
66
67 ```html
68 {% if site.search %}
69 <button id="search-trigger" data-index="{{ site.search }}">…</button>
70 {% endif %}
71 ```
72
73 The same rule as `site.feed`: the `if` is not defensive, it is how a theme
74 works on sites that turned search off. This site's search button, the palette
75 and the footer link all sit behind it — and the URL comes from the build, not
76 from the template gluing `search-` to a language code.
77
78 ## Ranking
79
80 Every term in the query must appear somewhere in a row — an AND, not an OR.
81 A hit in the title outranks one in the description, which outranks one in the
82 body, and a title that *starts* with the term outranks one that merely contains
83 it. Results are capped at twelve, which is as many as anyone reads.
84
85 Code blocks are indexed, because in documentation people search for the command
86 they half-remember.
87
88 ## The row schema
89
90 The client is yours to build, so here is what it reads. One row per page:
91
92 | Key | | |
93 |---|---|---|
94 | `t` | title | always present |
95 | `d` | description | omitted when empty |
96 | `u` | URL | always present, and it carries the subpath if you have one |
97 | `s` | parent section title | omitted when empty, and on section index pages |
98 | `c` | body plain text, code blocks included | always present |
99
100 Keys are one character because they repeat on every row. `"title"` instead of
101 `"t"` costs a few tens of kilobytes across a few hundred pages, for a file
102 nobody reads by hand.
103
104 ## The client
105
106 `search.js` on this site is a couple of hundred lines and has no dependencies. The dialog
107 is a `<dialog>`, so Escape, the backdrop, the focus trap and returning focus to
108 the trigger are the platform's behaviour rather than code. Open it with
109 `Ctrl`/`⌘` + `/`.
110
111 It is example code, not a feature of the tool: sqzass writes the index, and
112 what you build on top of it is yours.