A preview environment that lies to you

This came out of a stretch of work my team and I did on our preview environments, and honestly it taught me more about trust than about infrastructure.

If you haven't built these before, the idea is nice and simple. You open a pull request, and the system gives you a working version of the app on its own URL. No staging queue, no waiting your turn. It's live the moment the code lands. Anyone can click the link and see your change running for real.

We loved it. For about a month.

Then the small problems started showing up. Not crashes, nothing dramatic. Just little moments where the preview said one thing and the truth was another. And I learned something the hard way: a preview that's a little bit wrong is worse than no preview at all. When something is obviously broken, people work around it. When something looks fine but isn't, people believe it. They make decisions on it. That's the dangerous kind.

So I spent a while going through ours and fixing the spots where it was quietly lying. Here's what I found.

The previews were sharing a database

This was the one that scared me. Each preview is supposed to live in its own little world with its own schema. But our migration code would set up the right schema and then go and reuse a database connection it had cached from an earlier run. So a migration meant for one preview could end up running against another one's data.

Nobody noticed for a while, which is exactly why it bothered me so much.

The fix turned out to be tiny. Throw away the cached connection whenever the schema changes, and let it build a fresh one.

if (schema !== DEFAULT_SCHEMA) {
  process.env['DB_SCHEMA'] = schema;
} else {
  delete process.env['DB_SCHEMA'];
}

await ClearDatabasePoolCache();

One small assumption, that the cached connection was the right connection, and every preview was at risk. That's how these things usually go.

"It worked" wasn't actually checking anything

When a preview deploys, we tell our router to pick up the new route. The deploy was sending that instruction, getting back a "yep, got it," and calling the whole thing done.

But "got it" isn't "done." The router could fail to apply the change, and our deploy would still go green and happy. Meanwhile the preview URL led nowhere.

So now the sync actually writes down whether it worked, and the deploy reads that back and fails if it didn't.

status=$?
if [ "$status" -eq 0 ]; then result=success; else result=failed; fi

It looks almost too obvious written out like that. But a green checkmark finally means what everyone already assumed it meant.

A few smaller things, while I was in there

Once you start pulling on a thread like this, you find the neighbors. The script that updates the router was leaving temp files lying around forever, so I had it sweep out anything older than an hour.

find "$tmp_dir" -maxdepth 1 -type f -mmin +60 -delete

It was also assuming a particular tool was installed to hash some names, and when that tool wasn't there it quietly produced garbage. So I gave it a few alternatives to try in order before giving up.

Small stuff. But this is the kind of small stuff that decides whether you trust the system when you're tired and in a hurry.

The link in the comment was just a guess

This one is almost embarrassing. When a preview is ready, we post a comment with the URL. That comment had a single hardcoded link to one of our apps. Which is fine, until your change is for a different app, and now the comment is pointing people at the wrong place.

So I made it look at what you actually changed and link to the right app.

const hasWork = services.includes('work');
if (hasWork) body += `| Work App | ${workUrl} |\n`;

A wrong link in an automated message feels minor, but it does real damage. People stop believing any of it.

It was deploying everything, even for nothing

The last one surprised me. Someone opened a pull request that only touched test files. No real code, nothing that runs in production. And the system decided a whole pile of services had "changed" and tried to spin all of them up, to the point where it hit a hard limit on how much we're allowed to deploy at once. A pull request that changed nothing real couldn't deploy.

The fix was to ignore test files when working out what actually changed.

RUNTIME_CHANGED_FILES=$(echo "$CHANGED_FILES" \
  | grep -Ev '(test|spec|benchmark)\.[cm]?[jt]sx?$' || true)

If it can't change how the app behaves, it shouldn't be standing up infrastructure.

What I took away from it

When I look back at all of these, none of them were clever bugs. There was no exotic race condition I had to be smart to find. Every single one was just an assumption I'd made and never bothered to check. The cache is right. The instruction worked. The link is correct. The change matters. All reasonable, all wrong.

Preview environments are worth it. I'd build them again without hesitating. But they only earn their keep for as long as people believe them, and that belief is fragile. The real work isn't standing them up. It's making sure they never tell you something that isn't true.

Go check your assumptions. They're probably lying to you too.