Let’s say you want to know whether your business will work or not before you bet the farm on it.
But perfect information takes forever to collect. It’s just not worth it.
If your taco cart is successful, does that mean you can justify the 200k lease and kitchen refurb for your taco restaurant? Maybe, maybe not. But we need to build the whole dang restaurant to be totally sure. So we have to make the call whether to trust the quick & dirty version and commit (or walk away), or whether we want to spend more cash and time on another taco related experiment (delicious).
Or say you plan to rely on facebook ads to get new users. For the funnel to work, you need to get clicks for less than $1 apiece. When you run the test campaign, you’re getting clicks at $4.50 each. Optimising the ads isn’t your immediate concern, but you want to know that it’s at least plausible to get scale that way in the future. So is a result 4.5x worse than you need good enough? It depends.
If you had heard that facebook campaigns tend to perform 10x better once they’ve been run through the ringer, then yeah. In theory you should be able to optimize your costs down to $0.45 per click. So you could argue that the currently bad performance is still good enough for you to move forward.
Might that go wrong? Absolutely. Is it worth spending the time on additional tests to get more accurate information right now? Depends.
These answers aren’t really clear cut.
There’s always too much to do. Part of your job is knowing how much evidence is good enough.