Suzanne Chartier

Something I keep running into with vibe coding: the agent makes a failing test pass by loosening it instead of fixing the code.

One example: it replaced a strict deep equality check with expect.anything() on an error payload. The test still exists, still runs, still passes. It just stopped asserting anything. The build stayed green for days until users hit silent null-pointer crashes.

The same can happen with CI workflows and migrations. If the agent can edit them, it can make the gate disappear.

One cheap guardrail I ve seen: a CODEOWNERS rule requiring human review on tests/, migrations/ and .github/workflows/, so any diff there needs approval no matter how green the build is.
I d love to hear how you handle this:

Two of mine: "Never edit requirements.txt by hand, it's generated with hashes." Added after an unpinned dependency jumped to a new version on deploy and caused an incident in production. "Any API change updates the docs, the MCP tool descriptions and the slash commands in the same step." Added after an audit against production found the three had drifted apart. Curious what's in yours, and what happened before you wrote it.

I build solo, with coding agents doing most of the typing. Going through this week's threads here, I noticed the bugs that cost me the most had one thing in common: nothing crashed, nothing logged, and no user would ever have reported them.

1. Tests that couldn't see the risk. My API's pytest suite runs on SQLite, which has no Postgres row-level security or triggers. Green every time. Now I check RLS by hand with psql, as anon and as a logged-in user, inside BEGIN...ROLLBACK.