Teardown: I Shipped a Button Test. Then We Kept Changing the Button.
Poker players building startups. Free, Monday and Friday.
On October 28, SlotMaps put a new sign-up button at the top of every game page. The read date was November 2.
Four days later, the button was gone. Swapped for a new one.
Three weeks on, I still can't tell you if any of them worked.
SlotMaps is my AI slot assistant for advantage players. You snap a photo of a machine and it tells you if there's a play. It's run by a team of 17 AI seats, with me as the human in the middle. Nothing broke here. The team kept working hard the whole time. That's the problem, and the fix is one line you can steal today.
The test
Most SlotMaps visitors come from search. They land on a page about one slot game, read it, and leave. The goal is to get them to make a free account.
The team's starting number: of 126 visits from search to a game page in one week, 4 reached the sign-up page. About 3 in 100.
So the growth seat ran a test in two steps:
- October 26: a free-account button lower on the page, above the paywall box. Every click on every sign-up button now logs which button got it.
- October 28: a one-line ask right under the byline, at the top. The copy promises a free account, nothing more.
The plan was written down. Read it seven days after launch. Top button against bottom button, and both against that 3-in-100 baseline. First look November 2. Full read in about two weeks.
That's a good test. One change, one number, one date.
Then everyone kept improving it
On October 29, I paused all 17 seats to wait out my Claude usage limit. They came back Saturday, October 31. Two days lost. Not the real story.
The real story is what happened next.
- November 1: the one-line link at the top became two tap buttons with questions on them: "Is it worth playing now?" and "What should I bet?" The team even kept the same label on both so the read "stays on one label."
- November 7: the team noticed that game-page visitors who reached the sign-up page weren't finishing it. So they added tracking to the sign-up form to find out why.
- November 8: the two buttons came out. In went a little question box: type what you want to know, and it carries your question into sign-up. New read date, seven days out.
Every one of those changes makes sense on its own. Each had a good reason behind it. But the first test needed seven quiet days, and it got four. The second version got a week. The third got its own fresh clock.
On November 11 the team's notes finally said the right thing: hands off the game pages until the question-box read. At that point, nobody had typed in the box yet. The last change of any kind landed November 12. I can't find a read of any of the three versions in the logs.
The sample was always tiny
Here's the part that makes me feel a little better, and a little dumber.
126 visits a week is not much. Say the new button pushed it from 4 sign-up visits to 6. On a chart that's a 50% jump. In real life it's two people.
Any poker player knows this one. If you switch up your whole game every few hundred hands, you never find out if any version of it was good. You just get a pile of short, noisy sessions that tell you nothing.
That's what a page gets when you change it every four days on 126 visits a week.
Steal this
Every test you launch gets four lines, written before it goes live:
- Who reads it. A name or a seat. Not "the team."
- When. A date on a calendar that keeps running when you're away.
- What kills it. The number that says "turn it off," plus the smallest sample you'll trust.
- Hands off until then. No "quick improvements" to the thing you're measuring. Write the better idea down. It goes live after the read, as the next test.
Line 4 is the one my team learned the slow way. If you can't keep your hands off it, you haven't launched a test. You've just started redecorating.
The ledger
- SlotMaps: $384.88 MRR from 13 paying (the build page on November 18). The deal I gave that team: at least one new paying member every month, or I shut it down.
- The button test: three versions since October 28. Reads on file: none.
- R2F: today is Day 30, our first real scorecard. The bar is 100 members, 10 weekly active builders and 25 builds. Yesterday it was 19 members, 1 weekly active builder and 3 member builds. The scorecard grades it today, out loud.
R2F is a community for poker players building startups, any way they build. What are you testing right now, and how long have you kept your hands off it? Tell me in the discussion thread. And if you're a poker player building something, join R2F before November 20 and you get the Founding badge. Two minutes.
— Art
Written with the R2F AI team. How that works.
Post it and get feedback from players who’ve been there.
Join before Nov 20 and your profile carries the ♠ Founding member badge.
Comments