README.md (11140B)
1 # oni
2
3 A static Git browser for `debuman`'s public repositories. The Rust generator,
4 `sqzer-stagit`, produces the repository catalog, commit history, diffs, file views,
5 branches, tags, rendered READMEs, and Atom feeds. It reimplements stagit's feature
6 set without invoking the C stagit executable.
7
8 This project maintains its own HTML templates and CSS. Local build artifacts and
9 repository snapshots are kept in `.cache/`.
10
11 The site is published at [oni.sqzer.com](https://oni.sqzer.com) through GitHub Pages.
12 The source repository is [debuman/oni](https://github.com/debuman/oni).
13
14 All project documentation, code comments, and docstrings are written in English.
15 Mirrored repository content and Unicode test fixtures retain their original text.
16
17 ## Running locally
18
19 For a fresh checkout, install the dependencies listed below and run these setup
20 commands from the project directory in WSL Ubuntu:
21
22 ```sh
23 cargo fetch --locked --manifest-path generator/Cargo.toml
24 python3 scripts/bootstrap_git.py
25 ```
26
27 Then build the initial repository snapshot in Windows PowerShell:
28
29 ```powershell
30 .\build.ps1
31 python .\scripts\serve.py
32 ```
33
34 Preview: <http://127.0.0.1:4174/>
35
36 For later rebuilds, use `.\build.ps1 -Offline` to reuse the local repository
37 snapshot. Omit `-Offline` to refresh the public repository list and Git data from
38 GitHub. The generator itself is always compiled offline from the locked local
39 dependency cache.
40
41 These setup commands require network access. The Git bootstrap verifies the pinned
42 upstream SHA-256 and installs under `.cache/secure-git/`; it does not replace system
43 Git. Builds select that installation and require Git 2.56.1 or newer.
44 Feed URLs use the local preview address by default. The build script accepts
45 `--base-url https://oni.sqzer.com` when generating files for that hostname.
46
47 ## CI and deployment
48
49 [CI and deploy](https://github.com/debuman/oni/actions/workflows/deploy.yml) runs on
50 pull requests to `main`, pushes to `main`, and manual workflow dispatches.
51
52 - Every run tests Rust, generator behavior, security boundaries, and deployment
53 packaging, then builds all public repositories and checks the generated site.
54 - A successful push to `main` deploys the verified artifact to GitHub Pages.
55 Manual runs on `main` also deploy. Pull requests never deploy.
56 - Repository snapshots are fetched fresh. Changes made only in other repositories
57 appear on the next build; use **Run workflow** to refresh them without a source
58 change in this repository.
59 - The build job has read-only repository permissions. Only the separate deployment
60 job can write to Pages and request an OIDC token. Action versions are pinned to
61 commit hashes, and checkout does not persist credentials.
62 - The CI runner uses Ubuntu 26.04, Python 3.14, Rust 1.99.1, and the checksum-pinned
63 local Git build. Cargo dependencies and the patched Git installation are cached.
64
65 GitHub Pages is configured for GitHub Actions and the custom domain `oni.sqzer.com`,
66 with HTTPS enforced. The DNS CNAME points `oni.sqzer.com` to `debuman.github.io`.
67 Generated HTML and repository archives are deployment artifacts, not source commits.
68 The packager preserves dot-prefixed repositories such as `.github` while rejecting
69 actual Git metadata directories, symlinks, and special files.
70
71 The hosted site retains HTML sanitization, meta CSP, and referrer policy. GitHub
72 Pages does not apply this project's custom HTTP response headers; see
73 [SECURITY.md](SECURITY.md#hosting-requirements-and-remaining-boundaries) for that
74 hosting boundary.
75
76 ## Structure and dependencies
77
78 - `generator/`: Rust generator with a locked Ammonia HTML5 sanitizer dependency.
79 - `generator/src/repo.rs`: Repository access through the Git CLI. Repository code is not executed.
80 - `generator/src/html.rs`: HTML documents, navigation, catalog, and shared layout.
81 - `generator/src/main.rs`: File, commit, diff, reference, and feed generation.
82 - `generator/src/markdown.rs`: Rust bindings to the system MD4C library's public API.
83 - `theme/style.css`: Locally maintained CSS with no JavaScript, web fonts, or CDN dependencies.
84 - `scripts/build.py`: GitHub API requests, bare repository synchronization, compilation, generation, and archives.
85 - `scripts/readme_assets.py`: Relative README image and file link resolution.
86 - `scripts/benchmark.py`: Scaling benchmarks and output equivalence checks.
87 - `scripts/package_site.py`: Validated GitHub Pages artifact packaging.
88 - `.github/workflows/deploy.yml`: CI checks and deployment to GitHub Pages.
89 - `security/csp.txt`: Shared browser content policy for generated HTML and preview responses.
90 - `SECURITY.md`: Trust boundaries, security review findings, limits, and deployment requirements.
91 - `public/`: Generated site, including license notices under `licenses/`.
92 - `.cache/repos/`: Local Git inputs with remote push URLs disabled.
93 - `.cache/cargo/`: Compilation output.
94 - `.cache/previous-*/`: Previous local previews.
95
96 Builds run in WSL Ubuntu and require Rust/Cargo, Git, Python 3, a system linker,
97 `libmd4c-dev`, `libmd4c-html0-dev`, and `prlimit` from util-linux. The Git bootstrap
98 also needs make, a C compiler, zlib/OpenSSL headers, and curl development headers
99 (downloaded locally from Ubuntu when absent). Running the generator requires Git and
100 `libmd4c-html`. The generated site contains static HTML, CSS, and images, so the
101 hosting server does not need to run the generator.
102
103 MD4C handles Markdown parsing. Page generation and styles are maintained in Rust
104 and CSS; no separate C page generator or libgit2 is used.
105 Ammonia 4.2.4 sanitizes the resulting HTML with an HTML5 parser. It is the only
106 direct Cargo dependency; `Cargo.lock` pins its transitive dependencies.
107
108 ## Behavior
109
110 - The catalog is sorted by the latest commit date.
111 - Log shows the latest 50 commits; detail pages cover all commits reachable from HEAD.
112 - Rename and copy detection recognizes files with identical content. Diff graphs preserve insertion/deletion proportions.
113 - Tags outside the default branch receive feed entries and commit detail pages.
114 - The README navigation link opens the rendered `README.html` page.
115 - README selection order is `README`, `README.md`, `README.txt`, `readme`, `readme.txt`, then `README.markdown`.
116 - The Files view's `file/README.md.html` shows the original text with line numbers.
117 - Raster images with expected file signatures are copied into `raw/`. Repository SVG
118 images use commit-pinned URLs on GitHub's separate raw-content origin.
119 - README HTML retains safe formatting, tables, details, tasks, and fenced code;
120 executable content, inline styles, forms, and unsafe URL schemes are removed.
121 - `archive.tar.gz` contains the default branch's HEAD source tree.
122 - Diffs for deleted files remain readable and do not link to nonexistent current file pages.
123 - Repositories made private or deleted are excluded from the next online build.
124 - The generator also supports empty repositories.
125
126 ## Verification
127
128 From the project directory in WSL:
129
130 ```sh
131 cargo test --offline --locked --manifest-path generator/Cargo.toml --target-dir .cache/cargo
132 cargo build --release --offline --locked --manifest-path generator/Cargo.toml --target-dir .cache/cargo
133 python3 scripts/test_generator.py
134 python3 scripts/test_security.py
135 python3 scripts/test_package_site.py
136 python3 scripts/audit_site.py
137 ```
138
139 Tests create temporary local Git repositories to exercise Markdown, history
140 limits, complete commit pages, Unicode paths, binary and deleted files, renames
141 and copies, README selection, unmerged release tags, feed links, and empty
142 repositories. They also cover empty, mode-only, and merge diffs, navigation,
143 and README directory links. Stream tests cover marker-like filenames and source
144 lines, binary payloads larger than a pipe buffer, empty blobs, and child process
145 cleanup after success or failure.
146 Security tests cover independent XSS payloads, unsafe assets, metadata and output
147 symlinks, path traversal, size limits, preview host/origin checks, and HTTP headers.
148 Run `audit_site.py` after building to check every page, local link, feed, and archive.
149 The preview server requires Python 3.12 or newer.
150
151 ## Performance
152
153 Let F be the file count, C the generated commit count, and L the README link count.
154 For fixed filename lengths and path depths, the following repeated scans have
155 been removed. HTML output and Git's own diff computation costs are separate.
156
157 | Operation | Before | Current |
158 | --- | --- | --- |
159 | README, license, and submodule navigation selection | O(F × (F + C)) | O(F) setup, then O(1) per page |
160 | README directory link lookup | O(F × L) | O(F) setup, then average O(1) per link |
161 | Catalog input loading | Full history and file tree per repository | Latest commit only per repository |
162
163 Commit numstat and patch data are calculated together, eliminating duplicate diff
164 calculations. A persistent `git diff-tree --stdin` process handles all commits
165 in each repository, reducing diff process launches from C to one. Diff computation
166 still scales with the commits and their changes; replies are consumed one commit
167 at a time. Tags already covered by HEAD do not require another history query.
168
169 Output directories are prepared once per distinct parent directory instead of
170 checked for every page. README images share one lazy `git cat-file --batch`
171 process instead of launching Git for each image. Source lines are HTML-escaped
172 directly into the destination string, and patch rendering borrows its input buffer
173 without shifting the patch bytes. The build script also avoids copying the log
174 over an index page the generator has already written.
175
176 The benchmark script creates temporary repositories and checks that generated
177 files and catalog HTML are byte-identical between versions. It warms each workload
178 and reports the median of three runs by default. Compilation, network access,
179 and archive generation are excluded. Git process counts are recorded separately
180 from timing runs, and the optional asset comparison includes image extraction.
181
182 ```sh
183 python3 scripts/benchmark.py --baseline /path/to/previous/sqzer-stagit \
184 --baseline-assets /path/to/previous/readme_assets.py \
185 --report .cache/performance-scaling.json
186 ```
187
188 `--baseline-assets` is optional. Use `--sizes`, `--history`, `--images`, and
189 `--repeats` to adjust file, commit, and image counts and repetitions. Compare
190 binaries with the same site identity so the output equivalence check measures
191 implementation changes. Security sanitization intentionally changes HTML; do not
192 use a pre-sanitization binary as a byte-equivalence baseline.
193
194 ## Attribution and licenses
195
196 Project code is licensed under MIT; see `LICENSE`. Upstream stagit is a functional
197 reference, and its notice is retained in `third-party/stagit-LICENSE`.
198 MD4C's notice is retained in
199 `third-party/md4c-LICENSE`. License files are copied into the generated site;
200 no generator credit is displayed in the page footer.
201 Cargo dependency license texts are retained in `third-party/cargo-LICENSES.txt`;
202 regenerate them after lockfile changes with `python3 scripts/license_notices.py`.
203
204 - Upstream stagit: <https://codemadness.org/stagit.html>
205 - MD4C: <https://github.com/mity/md4c>