Dev notes

Telling Crawlers What’s Here

A full scan of this project turned up something plainer than a bug: the site had none of the five files a site is supposed to serve. No robots.txt, no sitemap.xml, no llms.txt, no security.txt, no web manifest. Around sixty write-ups sitting behind no map at all. The interesting part was not adding them, which takes an afternoon. It was deciding what to leave out, and stopping the sitemap from rotting the first time I add a page.

What was missing

Five files, none of them present, all of them standard:

robots.txt — nothing told a crawler which parts of a hundred-plus routes were worth its time.
sitemap.xml — around sixty write-ups, reachable only by following links from the home page.
llms.txt — the emerging convention for telling a language model what a site actually is.
/.well-known/security.txt — RFC 9116, a stated way to report something without guessing at an address.
A web manifest — the icons already existed as icon.tsx and apple-icon.tsx, so “Add to Home Screen” produced an untitled shortcut with nothing to name it.

The decisions worth writing down

robots.txt is a hint, not a fence. Disallowing /settings does advertise that it exists. That is fine: the session check is what protects it, and a crawler ignoring the file meets exactly the same check. The mistake would be treating the file as the protection.
No lastModified in the sitemap. The honest value is unknown, and a build timestamp on every URL claims every page changed whenever anything did. A crawler that learns the dates are noise stops trusting them, which is worse than saying nothing. Priority is omitted for the same reason.
The route list is a literal, and a test walks the filesystem. The sitemap runs in the Next runtime, where reading src/app is not something to depend on — so the list is source, and the drift is what gets tested. Adding a page without listing it fails a test.
display: "browser", not standalone. Standalone strips the address bar. This is a set of linked pages meant to be navigated and shared, and taking away back, forward, and the URL would make it worse rather than more app-like.
A contact that isn't an email. The repo is public, so security.txt points at GitHub's private advisory form. It reaches me and publishes nothing.

What I'd do now, and what I did

The five files a site is supposed to serve, after a scan found none of them present — and a proxy matcher that had been excluding /robots.txt for a file that was never written.
A sitemap of 102 URLs, 56 of them write-ups, which is the ratio that makes having one worth the trouble.
A test that walks src/app and fails when the sitemap list and the pages on disk disagree, checked by deleting a route and watching it go red — a guard nobody has seen fail is not a guard. It compares the two sets for equality, so it catches a page missing from the list and a listed route that no longer exists, in one assertion.
The same treatment for the other two files: security.txt fails thirty days before its Expires date lapses, and llms.txt fails when it names a write-up that has been renamed or removed. Both verified the same way, by breaking them on purpose and watching them go red.
metadataBase, which Next had been warning about: without it every relative metadata URL resolves against localhost.

Where it could go further

The route list is checked against the pages on disk, but nothing proves a page actually renders. A route that exists and throws is listed in the sitemap exactly like a healthy one.
llms.txt is still hand-written prose. The test catches a path that no longer resolves; it cannot tell that the description beside it has quietly stopped being true.
security.txt now fails the build thirty days before it lapses, which turns a silent expiry into a loud one — but somebody still has to move the date.

Next on this

A check that every sitemap URL returns 200, which the e2e suite is already positioned to do. That is the one thing here still resting on inspection rather than execution.

Telling Crawlers What’s Here

What was missing

Five files, none of them present, all of them standard:

robots.txt — nothing told a crawler which parts of a hundred-plus routes were worth its time.

sitemap.xml — around sixty write-ups, reachable only by following links from the home page.

llms.txt — the emerging convention for telling a language model what a site actually is.

/.well-known/security.txt — RFC 9116, a stated way to report something without guessing at an address.

A web manifest — the icons already existed as icon.tsx and apple-icon.tsx, so “Add to Home Screen” produced an untitled shortcut with nothing to name it.

The decisions worth writing down

robots.txt is a hint, not a fence. Disallowing /settings does advertise that it exists. That is fine: the session check is what protects it, and a crawler ignoring the file meets exactly the same check. The mistake would be treating the file as the protection.

No lastModified in the sitemap. The honest value is unknown, and a build timestamp on every URL claims every page changed whenever anything did. A crawler that learns the dates are noise stops trusting them, which is worse than saying nothing. Priority is omitted for the same reason.

The route list is a literal, and a test walks the filesystem. The sitemap runs in the Next runtime, where reading src/app is not something to depend on — so the list is source, and the drift is what gets tested. Adding a page without listing it fails a test.

display: "browser", not standalone. Standalone strips the address bar. This is a set of linked pages meant to be navigated and shared, and taking away back, forward, and the URL would make it worse rather than more app-like.

A contact that isn't an email. The repo is public, so security.txt points at GitHub's private advisory form. It reaches me and publishes nothing.

What I'd do now, and what I did

The five files a site is supposed to serve, after a scan found none of them present — and a proxy matcher that had been excluding /robots.txt for a file that was never written.
A sitemap of 102 URLs, 56 of them write-ups, which is the ratio that makes having one worth the trouble.
A test that walks src/app and fails when the sitemap list and the pages on disk disagree, checked by deleting a route and watching it go red — a guard nobody has seen fail is not a guard. It compares the two sets for equality, so it catches a page missing from the list and a listed route that no longer exists, in one assertion.
The same treatment for the other two files: security.txt fails thirty days before its Expires date lapses, and llms.txt fails when it names a write-up that has been renamed or removed. Both verified the same way, by breaking them on purpose and watching them go red.
metadataBase, which Next had been warning about: without it every relative metadata URL resolves against localhost.

Where it could go further

The route list is checked against the pages on disk, but nothing proves a page actually renders. A route that exists and throws is listed in the sitemap exactly like a healthy one.
llms.txt is still hand-written prose. The test catches a path that no longer resolves; it cannot tell that the description beside it has quietly stopped being true.
security.txt now fails the build thirty days before it lapses, which turns a silent expiry into a loud one — but somebody still has to move the date.