This site uses one functional cookie to keep feature rollouts consistent for you. Nothing is set until you choose. See the privacy notice.
Dev notes
I ran a security audit across both repos and expected to find bad code. I found almost none. What I found instead was a long list of things that were simply not there: a check on one route and not its sibling, a rate limit that looked configured and was not, a page confidently documenting behaviour nobody had run. Every single finding was an absence rather than a mistake, and once I noticed that, the research on AI-written code stopped reading like a warning and started reading like a description of my own diff.
Veracode gave the same 80 coding tasks to more than 100 large language models. Across all of them, 56% of the generated code passed basic security tests — and that number has not moved between report updates, despite a year of much better models.
The breakdown is the interesting part, because it is not uniform:
| Vulnerability | Pass rate |
|---|---|
| Insecure crypto algorithms | ~80–85% |
| SQL injection | ~80–85% |
| Log injection | ~12–13% |
| Cross-site scripting | ~15% |
Models are good at not writing the dangerous thing, and bad at writing the protective thing that was never asked for. SQL injection is avoided because the safe form — a parameterised query — is also the idiomatic form, so writing normal code produces safe code. XSS is different: avoiding it means adding an escape or a sanitiser that nothing in the prompt asked for, and nothing in the output looks wrong without.
Put plainly: the failure mode is omission, not error. Which means reviewing the diff is close to useless, because the diff is where the code is, and this is about code that is not anywhere.
These are real findings from the audit, all now fixed. I have grouped them by shape rather than severity, because the shape is the part that generalises.
/tables and /table/:tableName read information_schema and handed every table and column name to any caller with an account. There was a checkJwt on them, so they looked protected. Being signed in was never the question.
DELETE /markers/:id had the same shape from the other end: locations has no owner column and POST /markers is unauthenticated, so any signed-in user could delete anyone's marker. The route was removed rather than fixed — markers are create-and-read now.
Why it slipped: a middleware in the route definition reads as a protection. It answers “who are you”, and I read it as answering “may you”.
Posts respected visibility. Replies did not, so a thread served content the post route itself refused. Public profiles returned a private account's bio, counts, and the owner's Auth0 sub.
Why it slipped: the check existed and was correct. It just was not applied everywhere it needed to be, and nothing enumerates the places it needs to be. Both are one shared visibility rule now, so there is no second copy to drift.
A referral targetPath was validated with startsWith('/'). That accepts //evil.example — a protocol-relative URL — which is an open redirect on a link that looks like ours.
Why it slipped: it is validation. There is a check, it is about the right field, it reads correctly at a glance. This is the one I find least comfortable, because nothing about the line invites a second look.
The playoffs admin secret was compared with !==, which short-circuits on the first differing byte and turns guessing it into a per-character search rather than a search of the whole space. One shared constant-time compare now, replacing two near-identical copies.
Why it slipped: comparing two strings for equality is not a thing you look at twice, and the duplication meant there were two places to not look at.
The feature-flag resolver fell to the loosest access rung when a flag arrived from upstream without a tier on it. Absent was read as “no restriction” rather than “unknown”, so a flag could become more permissive by omission.
Why it slipped: the write-up listed it as a known rough edge rather than a bug, which is a very effective way to stop thinking about something.
Two of these, both the same shape. express-rate-limit dropped the max alias in v8, so every limiter in the API silently fell back to the library default of five per minute rather than its configured ceiling. Separately, one shared store instance across all limiters meant hitting one endpoint spent another's budget.
Why it slipped: the config object still had the option in it. A silently ignored option looks exactly like a respected one from the calling side, and nothing fails.
Four write-ups on this very site were wrong in the same direction — claiming a protection that was not on. /thoughts/security claimed frame-ancestors 'none' while the policy shipped 'self'. /thoughts/login-redirect described a timeout that armed prompt=consent so Auth0 would re-show the permission screen; it armed it, and it did nothing, so the page documented a timeout that silently signed you back in. /thoughts/improvements excused an in-memory rate limiter on the grounds that “the real defense is the backend” — and the backend's limiter was also in-memory, on a host that scales to zero, so every cold start wiped every counter.
Why it slipped: this is the one that genuinely bothers me. Writing the explanation felt like verifying it. A confident sentence about a protection is evidence of nothing except that I once believed it, and it actively suppresses the next person's instinct to check — including mine.
Reviewing harder does not work, because there is nothing to review. The things that did work were all some version of make the absence fail loudly:
Test the refusal, not the success. Most of the tests added by the audit assert that a request is denied — a non-owner gets 403, a private post 404s to a stranger, an unset allowlist locks everyone out rather than letting everyone in. A test that only proves the happy path passes just as well when the guard is deleted.
One implementation, not two correct copies. Two constant-time compares, two visibility checks, two rate-limit stores, two TLS configs, two connection pools. Every duplicate was somewhere a fix had landed on one copy. Consolidating is not tidiness, it removes the place where drift hides.
Make the config prove itself. The rate-limit bug is unfindable by reading and trivial to catch with one test that fires six requests and expects the sixth to survive. Anything read from configuration should have one test that fails if the configuration is ignored.
Treat absent as unknown, never as permitted. The flag resolver now refuses rather than relaxing. Unset allowlists mean nobody rather than everybody. When information is missing, the safe default is the restrictive one, and that has to be written down because the permissive default is usually the one that makes the demo work.
Since the audit, one more turned up, and it was not in the code at all. I asked an agent to work through some README tasks. To find my to-do list it read DATABASE_URL out of a local .env and queried the production database with it. Read-only, nothing damaged, and I only found out because I asked how it was reading the list.
It was not a jailbreak and it did not bypass authentication — it authenticated correctly, with a real credential I had left on disk. Going in through the database meant the Auth0 gate, the email allowlist and the owner re-check were simply not in the path. Postgres has never heard of an allowlist in Express.
The same shape as everything above: not a wrong thing done, a control that was not anywhere near where the action happened. The write-up on that lives at /thoughts/ai-security, including the uncomfortable part — the rule I wrote in response is prompt-level, and a rule in a prompt is advice rather than a boundary.
Update — August 14, 2026
The gap this page admitted to was that the audit was done by hand once, so the next absence would wait for the next time I went looking. Closing it meant turning the findings into checks that run on every push. What I did not expect was that the checks already there were part of the problem.
The operator write-up said the store detail page has seven tabs. It has eight; Restock History was added and the prose was not touched. A test asserted /seven tabs/ against that sentence — so when the eighth tab landed, the test went on guaranteeing the wrong number. It was pinned to the sentence rather than to the thing the sentence is about.
That is a different failure from the ones above. Every finding in the original audit was a control that was missing. This was a control that was present, green, and holding the mistake in place. Nobody reviews a passing test.
The world exhibits had the same shape from the other side: a test checked that every exhibit maps to a real feature, which is the direction that cannot fail. Nothing checked that every feature has an exhibit, so Research Explorer shipped without one while five places claimed the city held an exhibit for every feature. The llms.txt checks were identical — everything named exists, nothing about what was left out, which is the only way that file ever goes wrong.
The counts that had drifted were the cheap part to fix. The site was quoting 640 tests in two places while a third rendered the generated count of 2,525, a click apart. The changelog had stopped twenty versions back while the README called it a running log. Both now derive or fail.
The part worth keeping: I wrote a check for duplicate changelog headings that could never fail. Set.add() returns the Set, not a boolean, so my !seen.add(v) filter matched nothing and the test passed against a file I had deliberately corrupted. I only caught it because I had started corrupting files on purpose after the seven-tabs business. Rewritten, it immediately found a duplicate my own grep had missed.
So the rule I came away with is narrower than “write more tests”. Point the assertion at the data, not at the prose about the data. Check the direction that can actually fail. And watch the thing go red once before believing it — a guard nobody has seen fail is not yet a guard, and I now have two examples of my own to prove it.