โ† All posts

Railway Got Hacked. Here's What It Hit and My Plan B.

Wednesday was a dumpster fire for anybody running software on the internet.

A popular AI tool shipped an update with a burglar stuffed inside. Next.js, the framework this site runs on, slapped patches on two gaping holes. PHP had two of its own, and crooks were already crawling through them.

And then Railway got robbed.

What Railway is, and what happened

Railway is a hosting company. You shove your code at it, it runs your app. SlotMaps rents a room there.

To run your app, a host has to babysit your secrets: the Stripe key that charges customers, the database password, the AI keys, the email keys. All of it gets stuffed into what are called environment variables.

On Wednesday morning Railway caught someone rummaging through the systems that store those variables. Their words: the attacker "may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents."

They slammed the brakes on deployments for every customer, begged everyone to swap out every secret they'd stored there, and promised to email the accounts they confirm got hit.

The crooks didn't wait for the email. A security researcher watched about $371,000 in crypto get vacuumed out of wallet keys people had parked on Railway, starting a couple of hours before the announcement.

How it hit me

Here's the honest ledger.

  • SlotMaps' keys were in there. Stripe, the database, the AI keys, email, social accounts. Railway hasn't told me whether my account was in the "subset." Doesn't matter. I'm treating every one of them as stolen and torching them all.
  • SlotMaps is stuck in the mud. The site hasn't budged since Monday. My AI team had a security patch for SlotMaps written, reviewed and merged by 6 AM Thursday. It's gathering dust, waiting for Railway to flip deploys back on. I can mint new keys today, but I can't install them until Railway lets me deploy again.
  • Everything else dodged it. This site runs on Vercel. Snagworth runs on Cloudflare. PokerHQ moved to Vercel this week. Same night, my AI teams patched three live sites for the other holes while I mostly kibitzed. The fourth fix is the SlotMaps one, still gathering dust.

Time cost so far: one evening. Money cost: $0. That I know of. ๐Ÿ˜…

The poker part

Every online player who lived through 2011 knows this one. Don't park your whole roll on one site.

I'd been doing exactly that with my hosting. One company babysits the code, the database and every key. When that one site faceplants, so do I.

Plan B: Vercel

Step one happens no matter what: every key gets shredded and reissued at the source (Stripe, the AI companies, the email provider), the database password gets regenerated, and I comb each account for charges or usage I didn't make.

Step two depends on Railway. If deploys are still frozen for my plan by Monday, or Railway can't tell me what was actually taken, SlotMaps packs its bags.

The move looks like this:

  1. The app goes to Vercel, same as this site.
  2. The database goes to Neon, a hosted Postgres that PokerHQ already runs on.
  3. New keys go straight into the new home, never back into Railway.
  4. Railway stays up until the new one passes the same checks my AI team runs after every deploy. Then I flip the domain.

It's not free. It's a weekend of the robots' time and some of mine, plus whatever explodes that I didn't see coming. But a host that can't ship my security fixes isn't a host. It's a hostage situation.

If you're on Railway too: don't wait for their email. Rotate today.

What's your hosting setup? Would you have walked away from this one with your stack intact? Tell me in The Lobby.

โ€” Art

Written with the R2F AI team. How that works.

Building something like this?

Post it and get feedback from players whoโ€™ve been there.

Get one email a week

One email a week. Unsubscribe in one click.