This site uses one functional cookie to keep feature rollouts consistent for you. Nothing is set until you choose. See the privacy notice.
Dev notes
portfolio_api started as a pile of JavaScript route files. The overhaul turned it into a typed, layered TypeScript backend across twelve phases. This is the whole thought process behind it — what I chose, why, where I changed my mind, and why the decisions are sound system design even though (maybe especially because) I came at it as a frontend dev.
I am the person who calls this API all day. When a response shape is sloppy, or an endpoint 500s at 3am, or a field quietly changes type, the pain lands in the frontend. So the instinct that drove this whole thing is the same one that makes good frontend work: a boundary is a contract. A component's props are a contract. A hook's return type is a contract. An API's response body is a contract too — it just happens to cross a network instead of a function call.
Framed that way, the overhaul isn't really about "the backend." It's about making one system typed and predictable end to end, so the contract between the API and paul-explore is enforced by the compiler instead of by memory and hope.
Why it holds up
Backend and frontend aren't separate crafts here — they're two ends of the same contract. The skills transfer: thinking in typed boundaries, isolating what changes often from what doesn't, and measuring before optimizing are the same moves on both sides of the wire.
paul-explore talks to the API through BFF routes. Every existing endpoint path, request shape, and response shape had to stay stable the entire way through. That constraint is the reason this ran as twelve incremental phases instead of a rewrite. The old JavaScript kept serving live traffic while the new TypeScript grew beside it, and nothing got swapped until it matched byte for byte.
Why it holds up
This is the strangler fig pattern: wrap the old system, grow the replacement around it, and cut the original over only once the new path is proven. It's the professional answer to "how do you rewrite something that's in production" — you never have a scary big-bang cutover, every phase leaves the app shippable, and if a phase goes wrong you're one revert from safe. Small, reversible steps beat one heroic leap.
Added the TypeScript toolchain alongside the existing server.js rather than converting in place: strict mode, ES2022, NodeNext resolution, output to dist/. Then a layered directory — config/, middleware/, modules/ (one folder per feature), shared/ for errors, types, and utils.
The four layers have strict jobs: routes are a thin HTTP shell, controllers orchestrate, services hold pure business logic with no HTTP or DB awareness, repositories own all data access. On top of that: typed error classes (AppError and subclasses for 400/401/403/404/409/429) behind one global error handler, and a Zod-validated env object that crashes on boot if a required variable is missing.
Why it holds up
Two ideas a frontend dev already lives by. First, separation of concerns: routes/controllers/services/repositories is the same split as component / hook / API-client — the thing that renders shouldn't know how data is fetched, and the thing that fetches shouldn't know how it's rendered. Second, fail fast: validating env at startup means a missing secret crashes the deploy immediately with a clear message, instead of throwing a confusing 500 on the first user request an hour later. Cheap to catch early, expensive to catch late.
The most debatable decision in the whole project: three different data-access patterns, each isolated to the modules where it fits.
NbaRepository over the raw pg pool. Full control, zero abstraction tax, ideal when the queries are gnarly and you want to see exactly what hits the database..where() / .join() / transactions API. The sweet spot for lots of conditional CRUD where hand-writing SQL strings gets error-prone.Every remaining module (F1, fantasy, gallery, med-journal, feedback, chat, youtube, vitals, geo, google-auth, forum) got the same treatment with whichever pattern suited it — same response shapes throughout.
Why it holds up
Honest version: the variety is partly showcase. But it holds up because of the repository pattern and strict isolation. Every module hides its data tool behind a repository interface, so the service layer never knows whether it's raw SQL or Drizzle underneath — exactly like a frontend hook doesn't care if the data came from fetch, React Query, or a WebSocket. That boundary caps the cost of the variety: the complexity lives in one file per module and leaks nowhere. And it's honest engineering to keep this decision on trial — which is exactly what Phase 10 does.
Typed middleware for auth, validation (one generic validateBody<T> wrapper), rate limiting, and caching, with the Express Request type augmented globally so req.auth and req.validatedBody are typed everywhere. Then every console.log became structured pino logging — JSON in production, pretty in dev, child loggers per module, request-scoped correlation IDs.
Why it holds up
Logs are data, not strings. A console.log("user 5 did thing") is unsearchable; a structured { userId: 5, event: 'thing' } can be filtered and aggregated in a log platform. Correlation IDs let you follow one request across every log line it touches — the backend equivalent of tracing a single render through React DevTools. This is observability: you can't fix what you can't see, so you build the seeing-in first.
Explicit connection-pool settings with event listeners and slow-query warnings (>100ms), a /api/health check, and independent well-tuned pools per data-access pattern. The cache got redesigned into a typed manager with per-module TTLs (external API data 1h, aggregates 5m, RSS 15m, user-mutable data not cached at all), tag-based invalidation, and ETag / 304 support. Plus graceful shutdown: drain in-flight requests on SIGTERM, close every pool, flush the logger, kill Python children.
Why it holds up
The cache TTLs are the same judgment a frontend dev makes with a React Query staleTime: cache by how fast the data actually changes, not a blanket number. Graceful shutdown is deploy safety — Railway sends SIGTERM on every deploy, and without draining, in-flight requests die mid-flight and the user sees a random failure during an otherwise invisible deploy. Reliability is a feature; users just experience it as "it doesn't glitch."
Built response helpers (success, paginated, created) for a cleaner envelope shape — and then deliberately did not apply them to existing endpoints. Consolidated Zod validation per module with z.infer as the single source of truth, and generated an OpenAPI 3.1 spec plus Swagger UI from the route registry.
The pivot
The plan wanted every response wrapped in { data: ... }. But paul-explore expects raw bodies like res.json(teams), and wrapping them would break the contract on day one. So the "better" design got shelved for the live routes: legacy endpoints stay the raw v1 shape, only new endpoints use the envelope, and the difference is documented per controller. Discipline over aesthetics.
Why it holds up
This is the whole game: backward compatibility beats a prettier design. Wrapping the responses would have been cleaner in the abstract and a production incident in practice. Freezing v1 and versioning v2 is how you evolve a contract without a breaking change — the same reason you don't rename a prop half your components pass. OpenAPI then makes the contract machine-readable, so the frontend could generate types straight from the API instead of hand-writing them.
The moment everything pointed at: src/index.ts became the real entry point, mounting every new TypeScript router at the exact same paths the JavaScript used. Then the old routes/, middleware/, utils/, and server.js got deleted. That's the strangler fig finally cutting over.
The pivot
The delete was too aggressive. F1 and fantasy still lean on Python-queue plumbing (pythonQueue.js, queue.js, fantasyScoring.js) that was never ported to TypeScript, so the "clean sweep" broke them and took a geo cache import down with it. Those files had to be restored. The lesson: the Python-integration modules were far more entangled with JS than the plan assumed — porting them is real work, not a delete.
Why it holds up
Getting this wrong and recovering in one commit is the strangler fig earning its keep. Because the cutover was incremental and reversible, a bad assumption cost a restore, not an outage. That's the difference between "move fast and break things" and "move fast because things are cheap to un-break."
Moved to Vitest with supertest, test-data factories, and integration tests aimed squarely at the endpoints paul-explore leans on most — calendar CRUD, NBA stats with mocked external calls, vitals aggregation, profile uniqueness, post creation with mocked S3.
Why it holds up
The tests target the consumer contract, not a coverage number. An integration test that hits a real route through supertest and asserts the response shape is proof the frontend won't break — it tests behavior at the boundary that matters, the same reason frontend tests should assert what the user sees, not that a function was called. Coverage is a vanity metric; a green test on the exact endpoint you depend on is a safety net.
CI runs lint, type check, test, and build as parallel jobs. Knex migrations replaced hand-run SQL, with an initial baseline captured from init.sql. From here, schema changes are migrations, not manual edits.
Two things landed here later, both because writing the migration turned out to be the easy half. The container entrypoint now runs pnpm migrate before starting the app, so a deploy brings its own schema instead of relying on me remembering; a failed migration stops the server coming up and the previous release keeps serving. And a fifth CI job runs the migrations against a throwaway Postgres with only the environment declared in ci/migration-env.json, because until then nothing in this repo ran them at all and a migration that gained a requirement failed in the frontend repo instead.
Why it holds up
Migrations are version control for the database — an ordered, reviewable, reversible history of schema changes instead of someone SSHing in to run ad-hoc SQL. CI running on every push is the same seatbelt a frontend repo has: the contract can't regress without a red check, so "it works on my machine" stops being a deploy strategy.
Switched to pnpm for its content-addressable store (faster CI, smaller Docker layers) and, the real reason, strict node_modules resolution: you can only import what you actually declared, so phantom dependencies get caught instead of silently working until they don't.
Why it holds up
Same move, same reasoning as paul-explore's own npm-to-pnpm switch. Strictness surfaces latent bugs the flat node_modules hoisting hides. Making the implicit explicit is good engineering on either side of the stack.
A deliberate pass to confirm the design earns its complexity rather than adding to it, written up in plan/ARCHITECTURE_AUDIT.md. Checking that controllers stay thin, services stay free of HTTP/DB, and repositories own data access; flagging pure passthrough layers, single-implementation abstractions, and over-fitted utils. Only safe, mechanical fixes got made here; anything bigger was logged as a recommendation, not done on the spot. No new abstractions allowed in this phase.
Why it holds up
This is the maturity phase: knowing when to stop. Overengineering is as real a cost as underengineering — every needless abstraction is a tax the next reader pays. Putting the three-pattern decision back on trial, in writing, is the opposite of getting attached to your own cleverness. A frontend equivalent: auditing whether that context provider and four custom hooks actually earned their indirection, or whether one component would have been clearer.
Railway caps logs at 500/sec and we were tripping it (557 messages dropped per replica). The fix: ignore high-frequency zero-value paths (/api/health, /api/ready, /favicon.ico) entirely, and drop routine 2xx logs to debug in production so only warnings and errors flow at the info base level. A follow-up step measures real p95s before optimizing anything.
Why it holds up
Measure before you optimize. The log-rate breach was a measured problem with a targeted fix, not a guess. And the perf follow-up is explicitly "make it observable, then only touch what's actually over budget" — the same discipline as not reaching for useMemo until the profiler shows a real render cost. Premature optimization adds complexity to solve problems you don't have.
Closing the loop on the boundary violations Phase 10 surfaced. The posts controller was running raw BEGIN/COMMIT/ROLLBACK transactions inline — moved into a service. Google auth was calling DB helpers straight from route handlers — a service layer went in. The NBA repository held getCurrentSeason() date math that isn't data access — moved to the service. Behavior and response shapes stayed identical throughout.
Why it holds up
An audit you don't act on is theatre. Actually extracting the transaction logic and the date math puts the layers back where they belong, so the boundaries you drew in Phase 1 are real, not aspirational. Consistency across modules is what lets a new reader (or the next you) predict where any piece of logic lives — the same payoff as a component library where every component follows the same shape.
The plan was the intent; reality moved it in a few places, and writing that down keeps it honest. The through-line is the same every time: choose the option with the lower long-term cost, even if it means abandoning the original call.
NodeNext without the friction.express-rate-limit and Auth0's official express-oauth2-jwt-bearer instead of reinventing store/window and JWT logic. Don't hand-roll the security-critical parts.validateBody/Params/Query share one wrapper, so splitting them into files added indirection without value.apicache, jest, and ts-node got fully replaced but linger in package.json. Named honestly rather than pretended away.Strip away the specific tools and the same handful of principles show up in every phase:
The one thing the overhaul did not settle was what the frontend should do when this API is neither up nor down, but slow. The answer turned out to be “wait indefinitely”, which nobody chose — it was just what fetch() does when you give it no signal.
It surfaced when the API's own upstream, the NBA stats feed, stopped answering. The API stayed healthy: its health check returned in a tenth of a second while /api/nba/teams took 71 seconds to return a 500. The BFF route in front of it waited the whole time. Nothing rejected, so the page's error branch never ran, and a visitor got a team selector reading “Select a team…” with nothing in it. Not an error, not a spinner — a list that looked empty because it never arrived.
A survey found 47 unbounded call sites across 31 routes. Exactly one had got it right: the geo proxy, which already used an eight second AbortSignal.timeout. That became a shared helper and the number came with it, because a value that has already survived production is worth more than a fresh guess.
Two distinctions were worth preserving while doing it. A deadline now answers 504 and an unreachable backend answers 502, since slow and down are different operational facts and only one of them is worth waking up for. And a non-2xx response still passes straight through: a 404 from this API is something it genuinely told us, and rewriting that as a transport failure throws away the only real information in the exchange.
The wider point is about where a fallback stops helping. The whole design here is that the frontend degrades gracefully when the backend is unavailable, and that is right. But graceful degradation assumes a definite answer, even a bad one. A dependency that neither succeeds nor fails gives the fallback nothing to trigger on, and the interface ends up presenting “still waiting” as “nothing here”. A timeout is what converts an indefinite state into a definite one, which is the precondition for every other bit of resilience working at all.
None of this required a different brain than frontend work does. Typed boundaries, stable contracts, separation of concerns, caching by volatility, measuring before optimizing, and refusing to over-abstract are the exact instincts behind a well-built component tree — pointed at the other end of the wire. The overhaul is really one argument made twelve times: the frontend and the backend are a single system, and the same engineering judgment makes both of them good.
The most valuable habit wasn't any one pattern. It was keeping the old code serving traffic until the new code proved it matched — so the contract paul-explore depends on was never at risk, in any of the twelve phases.