When we are writing code for a product’s MVP, one of our core techniques is experimentation. We’ve found that frequent experimentation leads us to the quickest, most cost-effective way to dial in a product’s feature set.
Jeff Bezos, Founder of Amazon, says, “Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…”
Why do we experiment?
Many product development teams build software by defining a product’s entire feature set up front, without input from the customer.
We take a different approach. We measure the success of features with frequent experimentation. We are looking for information about whether or not changing a feature improved the customer’s experience with the product. Through this experimentation, we can quickly test risky features to see if the risk paid off and can adjust based on this information.
3 Reasons Experimentation is Important
We can save time and money building features. We build just enough of a feature to test the hypothesis that the customer will find it valuable. If it works, we can build out the rest of the functionality. If it doesn’t work, we avoided the cost of building out the whole feature unnecessarily.
Experiments provide measurable, actionable data. There is great value in following the intuition of the product developer. Experiments provide us a way to get beyond that intuition getting feedback and data to back it up. Having statistically significant data to show measured improvement is invaluable - and experiments can provide this data.
One of the biggest challenges we face in building software products is building the wrong features. There is a gap between committing to a feature and learning whether it’s right for the product. Often times a business won’t know if a feature adds value until months after it went live. We validate the feature as soon as possible so you know within a couple of weeks whether the feature is a keeper. This is why we work with tight feedback loops and have a culture of experimentation. It’s the best way to know we are headed in the right direction.
Experiments aren’t something you do only when you’re building an MVP or a prototype. It’s a process you can use at any stage of your product. I would argue that experimentation has a greater impact on mature products than in early stage MVPs.
Last month, Marc Hemeon mentioned this as his 3rd lesson learned in his announcement that they were closing their startup, Design Inc.
Each day, ship experiments. Especially for a SAAS product, there are always improvements to be made from onboarding to CTAs, the funnel can always be better. Establish a culture where teams are encouraged to ship experiments without feeling like they need permission and allow them to fail or succeed without consequence to their career.
In reading that post, you can tell that Mark regretted not encouraging people to experiment. All 4 points from his lessons learned reinforce how we approach prototyping and building out a product’s MVP.