# Testing The test suite lives in the [infra repository](development/developing) and runs against a theme checkout. See [Developing](development/developing) for how the two are wired together. ## Requirements **Node.js** at the version `.nvmrc` pins. Install it through a version manager - [nvm](https://github.com/nvm-sh/nvm), or [nvm-windows](https://github.com/coreybutler/nvm-windows) - rather than as a system package, so the version can follow the project rather than the machine. `nvm use` in the infra checkout reads that file, and so does CI, so the two cannot drift. The pin is not arbitrary. Before v26.8.1, Node's `fs.rmSync` silently removed nothing on Windows when a path contained a non-ASCII character, which made "replace this directory" quietly mean "merge into it" - and a suite whose whole job is comparing directories cannot live with that. **Hugo** at least the minimum the theme declares in its `theme.toml`. The plain edition is enough, as the theme uses no Sass. Install it the same way, through [hvm](https://github.com/jmooring/hvm), which keeps several versions side by side - the suite can then build against any of them, including the declared minimum, rather than only the one on your `PATH`. ## Running the Tests Check both repositories out side by side, then: ````shell cd hugo-theme-relearn-infra npm ci npm test ```` The parts run cheapest first and stop at the first failure, so a stale declaration fails at once rather than after a full round of Hugo builds. The whole suite is quick enough to run on every change. The checks are `tests/checks.js` and cover what the build layer cannot see: the runner's own configuration handling, the [dependency declaration](development/maintaining#sbom) - that it matches the vendored tree, and that every shipped component carries a license - and the properties the [SBOM](configuration/publishing/sbom) promises its readers, that it renders identically twice and that its serial number is recomputable from the document and moves with the contents. The SBOM comparison is `npm run sbom`, a narrower question: whether the committed `sbom.cdx.json` is still what that declaration renders. Each part also runs alone, and those with something to write back do it under an `:update` name: | Command | Runs | |---------|------| | `npm test` | everything | | `npm run test:update` | everything, rewriting whatever has something to write back | | `npm run format` | the formatting check | | `npm run format:update` | the formatting check, rewriting what it flags | | `npm run checks` | the runner checks | | `npm run sbom` | the SBOM comparison | | `npm run sbom:update` | the SBOM comparison, rewriting `sbom.cdx.json` | | `npm run golden` | the cases | | `npm run golden:update` | the cases, rewriting the expected output | `tests/golden.js` is the actual runner behind `golden`. Calling it directly takes the same flags and skips the rest, which is what you want while iterating on one case: ````shell npm test -- --build= node tests/golden.js --build= ```` Flags reach the runner directly; through `npm` they have to follow a `--` separator first, and land on the last command in the chain - so the first form still runs every part before the cases, and only then narrows. The examples below use the runner, being the shorter of the two. ## The Vocabulary A run is more than a list of sites, hence these words: | Term | Meaning | |------|---------| | Site | content plus the configuration it needs to be itself | | Axis | a dimension of configuration, each of whose values is a config directory | | Case | what to build, and how deeply to check it | | Build | one Hugo invocation - one site, one configuration | | Result | one output tree, compared as a whole | Most cases are one site, one configuration, one result. A case that varies an axis produces a result per combination. A case whose builds only mean something as a pair - a versioned site, or the docs and the exampleSite as GitHub Pages serves them - produces several builds sharing one result. Cases live in `tests/cases//case.toml` in the infra repository, and reading them is the quickest way to see what the suite covers. ## Shaping a Run The parameters, all optional: | Parameter | Selects | Default | |-----------|---------|---------| | `--build` | which builds to run | all of them | | `--hugo` | which Hugo to build with | each site's own, see below | | `--update` | rewrite the stored output instead of comparing against it | compare | ### `--build` - run part of the suite ````shell node tests/golden.js --build=minimal node tests/golden.js --build=url-permutations node tests/golden.js --build=url-permutations/urls-relative ```` `--build` matches a path prefix, so naming a case runs everything in it and naming a combination runs the one. A small case is cheap enough to run on every save while working on one thing. The accepted names are the ones a run prints. To see them without waiting for a full run, ask for something that does not exist and the runner lists them: ````shell node tests/golden.js --build=? ```` A sequence is the exception: its builds share one tree, so it is named as a whole and a prefix reaching inside it is rejected. A build lifted out of a sequence proves nothing, which is what makes it a sequence. ### `--hugo` - build with a particular version Left out, each site is built with the version an interactive shell would use in its own directory: the one its `.hvm` file names, or the `hugo` on your `PATH` when there is none. A pin therefore applies to the site it sits beside, and a run can legitimately span several versions. Each site a pin applies to says so in the output, so a result never looks like it came from a version it did not. Passing `--hugo` overrides every pin and holds the whole run to one version: ````shell node tests/golden.js --hugo=min node tests/golden.js --hugo=latest node tests/golden.js --hugo=v0.150.0 ```` `min` is whatever the theme declares in its `theme.toml`, and `latest` the newest release. Anything not already installed is fetched for you. Use `min` before pushing something that might rely on a newer Hugo feature, and `latest` to see a coming Hugo release before it reaches your users. ### `--update` - rewrite the expected output When a change legitimately alters what the theme produces, record the new output as the expectation: ````shell npm run golden:update node tests/golden.js --build= --update ```` A full regeneration also prunes: a stored result no case produces any more is deleted rather than left behind. A filtered run does not, having no way to know whether a result it did not build still exists. Commit the regenerated output together with the change that caused it, never as a commit of its own - otherwise the next person cannot tell which change produced which output. > [!warning] > The resulting diff **is** the test result. Read it before committing. An unreviewed regeneration turns the suite from a safety net into a rubber stamp. If the change spans both repositories, give both branches the same name; see [Developing](development/developing#working-across-both-repositories). ## Reading a Failure Every result is checked in layers, and the one that fails tells you what kind of problem you have. | Layer | Asserts | A failure usually means | |-------|---------|-------------------------| | Build | the build exits cleanly, with no unexpected `WARN` or `ERROR` | a template error, or a Hugo deprecation | | File set | exactly the expected files were generated | output formats, permalinks or a renamed page | | Content | every file is what was stored, byte for byte bar line endings and the checkout path | either a regression, or a change you meant to make | Layers are cumulative, and a case declares how deep to go with `layer`. It defaults to `content`, so a case opts down rather than up and always says why - the theme's own sites stop at the file set, because a content baseline over 2000 files would churn on every prose edit and be read by nobody. A pinned older Hugo reduces every case to the build layer, since Hugo legitimately changes what it emits between releases and a baseline holds for the version that produced it. The run says when that happened, so a build check never reads as a content check. ## Adding a Case First decide whether you need a new site at all. If an existing one already renders the thing you changed, extending its content is enough - add a page, regenerate, review the diff. ### A New Site Everything lives in the infra repository. 1. Create `tests/sites//` with a `config/_default/` and `content/`. Keep the configuration about the site - a title, output formats, content wiring. Nothing about reproducibility belongs there; that is what naming the `testing` environment does. 2. Write the least content that demonstrates your case. Sites are meant to stay small - a readable diff is the whole point, and one needing hundreds of pages is testing the wrong thing. 3. Add `tests/cases//case.toml`: ````toml site = "" environment = "testing" ```` 4. Generate its expected output, and read it: ````shell node tests/golden.js --build= --update ```` 5. Look at `tests/expected//`. This is the moment the case is worth something or not: if the output does not show the behaviour you set out to pin, it will not catch a regression in it either. 6. Commit the site, the case and the expected output together. ### Varying a Configuration Some behaviour only differs by configuration - URL generation being the standing example, where relative, absolute and ugly URLs are genuinely different paths through the theme. That is what an axis is for. Each value is a config directory under `tests/axes///`, and a case lists the values it wants: ````toml site = "url-permutations" environment = "testing" [axes] urls = ["relative", "absolute", "ugly"] ```` One content set, one result per mode, compared separately. Adding a further mode is a directory and one more name. An axis with a single value still applies - it just does not branch the tree, so nothing is nested that carries no information. ### Builds That Belong Together Some results are not one Hugo build. The published GitHub Pages site is the docs with the exampleSite beneath it. Neither half says anything alone. Such a case spells the sequence out, and its builds share one output tree: ````toml [[builds]] site = "docs@theme" environment = "github" [[builds]] site = "exampleSite@theme" environment = "github" dest = "exampleSite" ```` `dest` says where in the shared tree a build writes. The builds run in the order written, and the result is compared once, as a whole. ### Accepting a Known Warning Any `WARN` or `ERROR` fails a build unless it is listed in a baseline. These are consulted and their entries unioned: | File | Holds | |------|-------| | `tests/warnings.txt` | theme-wide, mostly Hugo deprecations | | `tests/sites//warnings.txt` | what a site's own content provokes | | `tests/cases//warnings.txt` | what a configuration provokes | A site's file is checked against the build of that site, so what the docs provoke applies wherever the docs are built. Each entry is a substring; a warning containing it is accepted. These baselines record outstanding work, not noise to be silenced. Adding an entry means consciously accepting a defect, so delete it as soon as the underlying issue is fixed and let a regression fail the suite again. ## Building a Site Outside the Suite A site that is not a case - a user's site attached to an issue, say - can still be built the way the suite would build it. Install the tool once as a global command: ````shell cd hugo-theme-relearn-infra npm install -g . ```` The global command links to the checkout, so it always runs the current state of the infra repository. Then run it from the site's directory, the one holding `content/`: ````shell test-hugo test-hugo min test-hugo 0.150.0 ```` The argument selects the Hugo version through hvm, defaulting to `latest`. The theme's `testing` configuration is layered onto the site's own, so the output is deterministic, and the result lands in a `public.+hugo.` directory inside the site. Beside the generated files it holds `metrics.log`, Hugo's log with template metrics, and `dir.log`, a list of every generated file - so two runs against different theme or Hugo versions diff directly. Nothing is compared or stored; reading the difference is up to you. ## Continuous Integration This repository runs the suite on every branch and every pull request, and nightly against the latest Hugo release - which is how a Hugo change that breaks the theme is found in CI rather than in an issue report. The infra repository runs nothing; one run tests the pair, and this is where it happens. That is why a change spanning both repositories is pushed to infra first, then here, and why a change to the suite alone has to be started by hand: see [Developing](development/developing#working-across-both-repositories). The suite never releases, deploys or publishes anything.