Engineering notes
from the field.

Every secret your CI pipeline touches is a secret at risk. The fix isn't a credential-free pipeline, that's impossible. It's a clean partition between what CI needs and what your app needs, short-lived credentials instead of long-lived ones, and a deploy path that never needs a shell into production.

A FastAPI app with default logging looks fine right up until your first production incident, when you discover it tells you what happened but not which user, which request, or in what order. Structured logging with bound context is a default you set up before writing your first route, not a refactor you do after.

An AI that says your database has reliability issues, with no source, no rule, nothing to point to, isn't a co-pilot. It's a magic 8-ball with a confidence problem. The fix is structured metadata that makes every recommendation traceable back to what actually produced it.

An ML model gives you a health score of 67 and no way to explain it. A rule engine gives you the same score and the exact list of checks that fired. For a system that influences engineering decisions, that difference isn't cosmetic, it's the whole product requirement.

It was accurate the day you drew it. Then three people changed the system and none of them updated the picture. That isn't a discipline problem. It's that the diagram is the only artifact in your stack with no consequences attached to being wrong.

JSONB and typed columns aren't competing strategies. They're tools for different access patterns. The right question isn't which is better, it's how you actually query the data.

A point lookup and a range scan against the same table have opposite indexing requirements. Building one table to serve both shapes optimises for neither. The fix is splitting the write model from the read model.

The migration worked perfectly in local Docker. On RDS, it failed with a missing extension error. After removing that line, it failed again. This time trying to drop an index that had never been created.

The application health check was green. Alembic was silently failing. Two drivers, one protocol, two different parameter names for the same setting. And a bug that passes every test until your first production deploy.

If your login endpoint takes measurably different amounts of time to respond to "email not found" vs "wrong password," an attacker can enumerate every valid email on your platform without ever seeing an error message.

The default on: push trigger fires CI and deploy simultaneously. If your tests take four minutes and your deploy takes two, broken code reaches production before the first test result comes in.

The layered structure every FastAPI tutorial teaches looks clean in a screenshot. It falls apart the first time you add soft deletes and have to touch five directories for one logical change.