This site uses one functional cookie to keep feature rollouts consistent for you. Nothing is set until you choose. See the privacy notice.
Dev notes
You should not run every test on every commit. Once a suite grows into the thousands, running all of it on every push burns feedback time and compute for no extra signal. The fix is to split tests by cost and run each tier where it pays for itself: fast unit tests on every push, heavier integration and end-to-end tests less often, and flaky ones quarantined so they never block a build.
Running the whole suite on every commit feels like the safe default, but it stops scaling the moment the suite gets large. Three costs grow together.
Speed. A long pipeline is a queue in front of every merge. When the only way to know if a change is good is a twenty-minute run, developers batch changes, context-switch away, and come back cold. Fast feedback is the whole point of CI, and heavy tests are what erode it.
Cost. Integration and end-to-end tests spin up real browsers, servers, and sometimes databases. Re-running that fleet on every push means paying cloud compute for work that mostly re-confirms what the last run already proved.
Signal. Most commits break something small and local — a type error, a null check, an off-by-one. Fast unit tests catch that class of bug in seconds. Reserving the expensive tiers for the changes and moments that actually need them keeps the fast signal fast.
Divide tests by how much they cost to run and how often that cost is worth paying. Each tier runs at a different cadence.
Run on every push and every pull request. They take seconds and exercise small, isolated pieces of code. This is the tier that gives instant feedback, so nothing heavier should ever sit in front of it. In this repo that's pnpm test (Vitest) alongside lint, typecheck, and the dead-code check.
Run on pull request merges or scheduled builds. They test how parts work together — a route plus its handler, a component plus its data layer — so they're slower than unit tests but still worth running regularly. Gating them on merge, rather than every push, keeps the per-push loop fast while still catching wiring problems before they reach the shared branch. In this repo they're the Vitest tests that render a component or route against its real data layer (react-query + MSW), tagged with a .integration.test.tsx suffix and run with pnpm test:integration.
Run nightly or right before a release. They drive full user flows in a real browser against a real environment, which makes them the slowest and most fragile tier. Running them on a schedule (or as a release gate) means the whole app gets exercised end to end regularly, without that cost landing on every individual commit. Here that's the Playwright public and authenticated projects, axe accessibility scans included, wired to a schedule trigger. Every push and PR still runs a thin @smoke-tagged subset so pull requests keep a real-browser signal without the full cost.
Isolate them so they can't break regular builds. A test that fails intermittently — a race, a timing assumption, an external dependency — trains everyone to ignore a red build, which quietly destroys the value of the whole suite. Move it to a background or on-demand job, keep the blocking gate deterministic, and fix the underlying flake on its own schedule.
The split lives in .github/workflows/ci.yml, as five jobs, each running as often as its cost is worth paying.
The quality job is the fast tier: it installs, lints, typechecks, runs the dead-code check, and runs the unit suite with pnpm test. It runs on every push and pull request to main and develop, and it's cheap enough to be the thing developers wait on.
The e2e-smoke job runs a thin real-browser subset on every push and PR. It declares needs: quality, so it never starts until the cheap checks are green, and it runs only the @smoke-tagged Playwright tests (--grep @smoke). PRs keep a fast end-to-end signal without paying for the whole suite.
The integration job is the middle tier: the Vitest tests that render a component or route against its real data layer (pnpm test:integration), gated on needs: quality. The textbook cadence would gate these on merge, but here's a deliberate deviation: this repo's integration tests run in milliseconds (Vitest + MSW, no real browser), so we keep them on every push and PR as well as nightly, rather than merge-only. When a tier is that cheap, the reason to defer it disappears — it's better to catch a wiring bug on the PR than after it lands on the shared branch. The cadence rule still bites where the cost is real: the genuinely slow end-to-end flows below are what move off the per-commit path.
The e2e-full job is the heavy tier: the full public and authenticated flows plus axe scans. It's gated behind needs: quality too, but it only runs on a nightly schedule or via workflow_dispatch before a release — never on an individual commit. Anything tagged @flaky is held off the blocking path with --grep-invert @flaky and re-run in a separate continue-on-error step, so a flaky test gets exercised nightly but can never fail a build.
The fifth is e2e-operator-live, and it is the odd one out because it is not really a tier of this repo at all. It stands up Postgres, checks out portfolio_api at main, applies its migrations, seeds it, starts it, and drives the operator flow against a real backend rather than a mocked one. It prefers an API branch of the same name when one exists, so a paired change can be tested as a pair.
That job was also where a whole class of breakage surfaced, and the reason is worth more than the fix. Because it applies the API's migrations, it is the only thing anywhere that ran them. The API repo itself never did. So when a migration gained a requirement, this workflow was the thing that noticed.
Concretely: a migration started encrypting stored OAuth tokens and refuses to run without TOKEN_ENCRYPTION_KEY. It was green in the API repo, because nothing there ran migrations, and it turned every branch of this repo red. The person reviewing that migration never saw a failure. The people who saw the failure were working on something unrelated in a different codebase.
What makes that hard to catch is that it is invisible to inspection. There is no process.env anywhere in the migration files. The requirement arrives transitively, through an import, so grepping for it comes back clean and the only thing that finds it is running them.
So the requirement is now declared rather than discovered. ci/migration-env.json in the API lists what the migrations are allowed to need, and a shared script runs them with exactly that and nothing else. Both repos call the same script against the same file, which is the part that makes it a contract instead of two lists that drift: a migration that gains a requirement now fails in the repo that added it, and declaring it fixes both sides at once.
The general shape, which I suspect is common in any multi-repo setup: if repo A depends on something in repo B, and only A's CI exercises it, then B's CI is not testing B. It is testing the part of B that nobody is going to break. The interesting failures were always going to land somewhere else.
Two things went wrong here that no amount of tier discipline would have caught, and both are worth more than the tiering itself.
The first is that a tier can be green for reasons unrelated to what it claims to check. A set of end-to-end specs targeted a seeded store id, and when a real backend was serving, the API returned a 404, the app fell back to its seed exactly as designed, and the specs passed identically whether or not the backend worked. A test that cannot fail when the thing it covers is broken is not a test. The fix was not a new tier; it was making the specs assert which backend they had actually reached.
The second is that a tier can quietly stop running. The accessibility specs waited on networkidle, which is a promise about whichever third party the page happens to call rather than about the page. When one of those upstreams stalled, two routes timed out, which reads as a slow test rather than an absent one. They had never actually run the scan. Replacing the wait with the page's own load, its main landmark and document.fonts.ready made axe run on them for the first time, and it immediately found real serious-impact contrast failures that had been shipping. The suite reported those routes as covered for as long as they were broken.
A third version is worse than either, because the tier does not run at all. Six stacked pull requests — each based on a feature branch rather than develop — got no CI, because the workflow only triggered on pull requests into main or develop. Every one of those PRs looked unchecked not because a test failed but because none were ever scheduled, and it took hand-triggering the workflow to notice. The suite was fine; the trigger simply excluded the PRs it most needed to gate. Dropping the base-branch filter fixed it — but the failure shape is the same: a PR page showing no red can mean “all checks passed” or “no checks ran,” and the two are indistinguishable until you ask which.
A fourth arrived on this very pipeline, and it was the plainest of all. The job that runs the operator screens against a real backend had been pinned to that backend's develop branch, when the deployed dashboard only ever talks to its main. The pin was wrong twice over: it tested against unreleased backend work the running app never sees, and it tied the job to a branch that a develop-to-main release merge auto-deletes. So the first nightly after a backend release failed at the checkout, fetching a develop that no longer existed — and it had looked fine until then only because the one earlier run predated the release that removed the branch. The fix was a single word, develop to main: test against what is actually deployed, which is also the one branch a release cannot delete.
All four have the same moral, and it is not about cost or cadence: a passing tier is a claim, and so is an empty one. It is worth occasionally checking what a green run actually exercised, because the failure mode is not a red build — it is that you stop looking.
A week of this turned up four separate versions of one failure, and none of them showed up as a red build. Every one was a suite passing while measuring something other than what it claimed.
What ties them together is that a test run against the wrong inputs does not fail. It passes, which is the only outcome nobody investigates. Tiering is about spending test time where it pays; nothing in that idea protects you from measuring the wrong thing carefully.
There was a fifth, and it is the one I had been walking past. A full run carried thirty React warnings about state updated outside a test's control, forty-five lines about a missing canvas, and forty-odd unmatched network calls. I had been reading all of it as noise. It was not: the warnings were the visible half of two endpoints that had no mock at all, so every component touching them updated state after the test had finished; the canvas lines came from an assertion that would have thrown in any browser without 2d support; and one of the unmatched calls was a test reaching for a public blockchain node on the open internet, prevented from leaving the machine only because the mocker happens to reject anything unmatched.
Noise is not a category of output. It is a decision to stop reading, and it is made once and then held. The unmatched mocks that cost days to find were sitting in that same stream the whole time. Everything here now runs clean, so the next unexpected line is worth looking at.
So each one now has a guard that fails loudly rather than a note asking people to remember. Mock registrations are asserted against the source file, since the symptom is invisible in results. The installed package version is checked against the manifest, with a message naming the fix. Live-mode browser runs assert they reached a real database rather than a fixture. Each guard was checked by making it fail on purpose first — a guard nobody has seen fail is just another claim.
Match test cost to how often the answer changes. Cheap tests that catch the common failures run constantly. Expensive tests that catch rare, system-wide failures run rarely — but they still run. Flaky tests run nowhere near the blocking path until they're fixed. The goal isn't to run fewer tests overall; it's to make the pipeline a fast, trustworthy signal instead of a slow, expensive tax on every commit.