Taking the database off the public internet

What I found when I looked

DATABASE_URL pointed at switchyard.proxy.rlwy.net, which resolves to a routable address. Not a private network, not a VPC — a hostname on the public internet with Postgres behind it. The only control on that path was the password.

Nothing was wrong in the sense of misconfigured. That is the default a managed provider gives you, because it is the one that works from your laptop on day one, and it keeps working, which is exactly why you stop thinking about it. The bill for that convenience is that every application-level control I had built — the Auth0 gate, the email allowlist, the owner checks — sits in Express, and Postgres has never heard of any of them. Anyone holding that one string skips all of it.

I know that shape from the front end. It is the same reason you cannot trust a disabled button. The control has to live where the action happens, and mine did not.

The TLS rabbit hole, and why I stopped digging

The obvious first move was to verify the server certificate, so the connection at least proves it is talking to the right database. I spent a while on it and the answer was no.

Railway's proxy presents a certificate with CN=localhost, signed by a private CA named root-ca. So the system trust store rejects it as self-signed. Supplying their CA and keeping hostname checking on still fails, because localhost is not switchyard.proxy.rlwy.net. The only combination that connects is their CA with hostname verification turned off — and they do not publish that CA, so it has to be scraped out of the handshake.

I nearly did it anyway. Pinning a scraped certificate would have felt like progress. It would also have traded a real security control for a future outage, because the provider can rotate that certificate whenever they like and my pin would break the connection with no warning and no obvious cause.

The decision I wrote down was: do not pin. Not because verification does not matter, but because I was about to spend real effort making a connection trustworthy that should not have been crossing the internet in the first place. The fix was one layer down from where I was looking.

Private networking, which makes the question moot

Services in one Railway project can talk over a private network — a WireGuard tunnel, with DNS names like postgres.railway.internal. Point the API at that instead of the proxy and the internet path stops existing. Verification is not solved, it is irrelevant, which is a much better place to be than a correctly verified public connection.

Two things about it are worth knowing before you try, because both would have bitten me:

It only exists at runtime. The private network is not available during the build phase. If migrations had been part of my build step, this switch would have broken every deploy. Mine run from the container entrypoint, which is on the right side of that line — by luck rather than judgement, since I had put them there for an unrelated reason a few hours earlier.

Older environments are IPv6-only. Private DNS resolves to IPv6 addresses on environments created before October 2025. Node handled it without complaint, but it is the first thing to suspect if a connection refuses.

On the way there: nothing ran the migrations

I went looking for where migrations run in production and the answer was nowhere. railway.json starts the process, the Dockerfile starts the process, and neither touches knex. The only pnpm migrate anywhere was in the frontend repo's CI, against a throwaway database it creates for end-to-end tests.

I had written down, in an earlier note on this site, a considered argument for keeping migrations manual: two deploys could race, a destructive migration could go out before I had read it, and a half-applied migration leaves a broken release. Keeping it manual, I said, cost one command and kept the ordering something I owned.

What changed my mind was watching the cost land. I added a migration, shipped it, and only noticed while writing the release notes that nothing would ever apply it. “One command” is only one command if you remember it, and a step that lives in your head has no failure mode — it just has you.

Taking my own three objections seriously rather than pretending I had been wrong: the racing one was answerable, since knex takes a lock and the real risk was cron containers sharing the image, which now skip the step. The half-applied one turned out never to have been a risk at all — Postgres has transactional DDL and knex wraps the batch, so a migration that throws leaves nothing behind. I had listed it as a cost of automating without ever checking whether it was a property of the tool I was already using.

The third objection stands. Automating when migrations run does nothing about what is in them. So that one is gated rather than argued away: a test reads the up() of every migration, looks for drops, renames, truncations and column tightening, and fails unless the file writes down why.

// DESTRUCTIVE: drops todos.detail, unused since 4.9.0 and
// confirmed empty in production before this shipped.

An acknowledgement rather than a ban. Dropping a column is sometimes exactly right; doing it without having thought about the code currently running against that schema is not. Writing the reason costs nothing when you have thought about it and is impossible to produce when you have not.

Waiting, but not forever

One more thing had to go in before the switch. The private network is not up the instant a container starts, so a migration firing immediately can fail on DNS or a refused connection — and because the entrypoint runs under set -e, that is a failed deploy which reads as the migration broke when the network simply was not ready. The worst kind of error: it sends you to look at the wrong thing.

So the entrypoint polls before it migrates, bounded at about thirty seconds. Bounded is the whole design. Retrying forever turns “the database is gone” into a container that never starts and never says why, which is harder to diagnose than a clean failure.

It did not fire on the day. The database answered on the first attempt, so the retry never ran, and that is the right outcome rather than a wasted change — the failure it guards is intermittent, and you find out you needed it on the deploy where you did not have it.

The switch, and what the logs told me

Repointing was one variable: a reference to the Postgres service's private URL rather than a pasted proxy string, so it tracks if the service is ever recreated. The deploy came up, migrations reported nothing pending, the app served.

The logs then told me something I had not gone looking for. They reported env: "development" — in production. NODE_ENV was simply never set, and the config defaults to development when it is missing.

That mattered more than it looks, because of one branch in the TLS helper:

if (opts.nodeEnv === 'production') return { rejectUnauthorized: false };
return false;   // ← TLS off entirely

Everything I had written about this connection — including the decision above — described it as TLS without verification. It was not. With NODE_ENV unset, it was no TLS at all, over the public internet, for as long as that had been true. It also explains why the boot warning I had added about unverified connections never appeared in any log: it was gated on the same condition.

The consolation is that the change I was already making closed it, because traffic on the private network is encrypted by the tunnel. I fixed a worse problem than the one I set out to fix, and I only found out about it because I read the deploy logs line by line instead of checking that the deploy was green.

Then I broke production for four minutes

Last step was rotating the password, which by then was overdue for reasons written up elsewhere. There is a Regenerate button. I pressed it, and the API went to dbConnected: false.

The mechanism is worth knowing, because it is a trap with a delay built into it. Setting the password variable only applies at first initialisation; on an existing database the variable and the actual role drift apart. And Postgres does not drop established connections when a role's password changes — so a running app carries on working perfectly on its existing pool. Everything looks fine. The failure arrives at the next restart, which might be days later, for reasons that look unconnected to anything you did.

Mine failed immediately only because the platform restarted the container as part of the change. That was luck, and it was the good kind: a four-minute outage I could see and fix is enormously better than a landmine sitting in the deploy pipeline for a fortnight.

Recovery was a redeploy so the app picked up the regenerated credentials. The lesson I actually take from it is not about Postgres — it is that I checked the wrong thing first. My instinct was to look at the dashboard panel that was showing an error. The useful signal was the health endpoint, because that is the one that tells you whether the thing users touch is working.

One pool, afterwards

With the connection path settled I could finally do a cleanup I had been deferring. The app built two connection pools: one in the CommonJS layer that predates the TypeScript rewrite, one in the TypeScript config. Two pools means two sets of connections to the same database and, the part that actually bit, two places to configure TLS, which had already drifted apart once.

I had deliberately not done it earlier, and the reason is the part worth keeping: the test suite mocks the pool, so nothing in it could have caught a break, and the blast radius ran through the Google Calendar sync path. A change I could not verify, whose failure would be silent and in production, is not a change to make because the code looks untidy.

What made it safe was not courage, it was having a way to check. Both entry points now resolve to the same object, which I proved by comparing identity rather than by reading the imports; the built app boots against a real database; and the route that goes through the old CommonJS layer answers 401 rather than 500, which is precisely the failure I was worried about.

What I would tell another front-end dev

The default connection string is a product decision, not a recommendation. It is public because that is what works from your laptop on day one. It keeps working, which is why nobody revisits it.

Removing a path beats securing one. I spent hours on certificate verification for a connection that simply should not have existed. The fix was one layer down from where I was looking, and it made the hard problem irrelevant instead of solving it.

Read the deploy logs, not the deploy status. Green meant the container started. The line that mattered was four words in the middle saying the environment was development, and it had been true for months.

Do the scary step where you can see it fail. Credential rotation has a delayed failure mode by design. Force the restart yourself, immediately, while you still remember what you changed.

None of this was difficult. It was unfamiliar, which is a different thing, and the parts that took longest were the ones where I was solving the problem I recognised rather than the one in front of me.