This site uses one functional cookie to keep feature rollouts consistent for you. Nothing is set until you choose. See the privacy notice.
Dev notes
A week in the design system repo that turned up four green checks, none of which was measuring the thing it claimed to measure. A CI suite nothing ran, a visual gate with no quota left, a contrast figure taken under a layer that was hiding it, and a test count that changed depending on whether I had run a build.
The design system publishes @paul-portfolio/tokens, css and react to npm the moment something reaches main. There are 864 tests across four workspaces standing behind that, including the palette contrast gate and the component contrast pairs that the accessibility notes lean on. Until this week, nothing in CI ever ran them.
The workflows directory had three files in it — chromatic.yml, publish.yml and tag-release.yml — and not one of them invoked a test. The suite was green the entire time, which is true and misleading in the same breath: it was green because I remembered to run it before pushing. That is a habit, not a gate, and the difference only shows up on the day the habit lapses.
The fix is unremarkable, one ci.yml running lint, typecheck, build and the suite on pull requests and on main. What I want to record is the step before merging it. A workflow that has never failed has not been shown to work — it has been shown to run, which is a much weaker claim and looks identical in the checks list. So I gave it something real to catch: flipping chart palette slot 2 back to the violet it used to be reproduced a genuine deuteranopia collision, and the job came back with expected 11.35 to be greater than or equal to 15 and exit 1. Then I put the palette back and merged the red-proven workflow.
Chromatic was the one gate that had been wired up correctly for months, and it is the one I trusted most, because a visual diff catches the class of regression no assertion is ever going to describe in advance. I opened it for a completely unrelated reason: I was working out whether to move that workflow to Node 24 along with everything else.
The UI Tests check reads Update your plan to resume testing. The snapshot quota has been exhausted for about a month. The chromatic job next to it reports pass, because exitZeroOnChanges: true is set and a build that could not take a snapshot has, technically, no changes to report. It was green on a flag, not on a comparison, and nothing about the checks list distinguishes those two things.
The sharper half is the setting underneath it. autoAcceptChanges: main means a build on the release branch takes whatever it sees and makes it the reference for everything after. So if any drift did land during that month — and the recolour work means it plausibly did — the gate would adopt it as the new baseline on the next release merge. A check that misses a regression is a gap. A check positioned to ratify one and then vouch for it afterwards is worse than not having it, because it launders the change.
So the Node 24 question answered itself, in the other direction from the one I expected. Everything else moved; Chromatic stays exactly where it is until there is quota to prove the move changed nothing. An unverifiable visual change riding along inside a CI-configuration pull request is the trade I would least like to explain later.
Version 0.2.36 recorded .btn--gel as a known AA failure at 3.45:1 and deferred it, which is a reasonable thing to do with a decorative variant when the number is close. The number reproduces exactly, every time. It is also the wrong measurement, and the two facts together are what let it sit there: a figure that reproduces feels settled.
It reads white against the bare primary-500 gradient stop. That stop is real, but it is not what anyone sees, because a 55% white gloss sits on top of it. The figure was measured under the gloss rather than through it. Composited properly, the floor was 1.69:1 at rest and 1.52:1 on hover — understated by more than half, and understated in the direction that lets something keep shipping. I would rather have found this as a failure than as a correction, but a defect recorded at half its size is a defect nobody prioritises.
Reframing it also decided the fix, which the original framing could not have. My instinct was to darken the ramp until the ratio cleared, and that was arithmetically impossible from the start: a 55% white gloss caps whatever is underneath it at 3.35:1 even over pure black, and AA wants 4.5. No amount of darkening reaches it while the gloss stands. The gloss had to come down with the ramp. Shipped primary-700 to primary-900 under a 14% gloss, which measures 5.12:1 at rest and 4.73:1 on hover at the worst stop along the fill.
The root cause was the sampler, and it is the reason this belongs with the other three. It read the discrete ramp steps that a background names. A gradient’s interpolated midpoint has no name, and a translucent layer above the fill has no name either, so neither was visible to it — and both blind spots err toward passes, which is the only direction that matters. The replacement composites every half percent along the fill and measures there.
It also carries a guard on itself, which is the part I would keep if I kept one thing. The composited floor has to stay strictly below the bare-stop reading, because compositing a white gloss over a fill can only ever reduce contrast against white text. If the compositing silently stopped happening — a refactor, a bad default, a layer dropped from the model — then ratios would improve while the button got worse, and the suite would report the improvement. That is the failure this whole page is about, so the new check is built to notice it happening to itself.
packages/tokens/tsconfig.json had include: ["src"] and no exclude, so tsc compiled the tests along with everything else into build/. Vitest ships a default exclude that covers **/dist/** and does not cover **/build/**. So 42 became 84 the moment a build had happened, and the extra 42 were compiled copies quite capable of passing while the sources they came from failed.
A test count that moves depending on whether you happened to build first is a small thing on its own. The consequence was not small: files: ["build/"] meant every published tokens tarball carried ten test files out to every consumer. Fixing the config took the package from thirty-one entries to twenty-one, which is a third of a published artifact that existed because a compiler was told to look in one place and nobody told it what to skip.
The tell was sitting one directory over the whole time. packages/react was covered twice against this — it sets an exclude and it emits to dist/, which vitest already ignores — where tokens was covered by neither. Two packages in one repo, one belt-and-braces and one bare, and the asymmetry is only visible if you read the two configs side by side, which nothing ever asks you to do.
Four checks, four different subsystems, one shape. Every one of them was green, and every one was measuring something other than the thing it named. The CI badge measured whether a workflow file parsed. The visual gate measured whether the Chromatic client exited cleanly. The contrast number measured a surface that was never rendered. The test count measured whatever files happened to be on disk. In each case there is a real signal available and a cheaper adjacent one, and the adjacent one is what got wired up — not through carelessness, but because it is the one that is easy to reach and it looks the same once it is green.
Not one of these was caught by a test failing. The CI gap turned up while adding an unrelated workflow. Chromatic turned up while pricing a Node upgrade. The gel button turned up while writing the contrast pairs for something else. The tokens count turned up because a number looked different than I remembered. Four for four, the trigger was opening a file that nothing was pointing me at.
I do not think that generalises into a practice, and I am wary of the version of this that becomes a rule about reading more files. What it does change is how I read a green check on something I did not wire up myself: the question is no longer whether it passes, it is what it would have to see in order to fail, and whether it has ever done it once. The workflow in the first section is the only one of the four that has answered that question out loud.
The same pull request collapsed the CI Node matrix to 24 alone — publishing already ships from 24, and Node 20 went end-of-life in April — which took three jobs down to two. That one is just housekeeping, and it is in here only because it is the change that had me looking at the workflows at all.