Tony Fadell on the pitfalls of A/B testing and how it's not product design
It’s become fashionable in recent years to use lots of small A/B tests to try to build and refine products and experiences — even though there is no evidence this actually works.
While A/B testing can be a powerful tool in the right hands, it can also be an incredibly dangerous tool in the wrong hands. And I think most product and tech teams don’t have the sophistication to pull off A/B testing properly.
I wrote last week that You can’t A/B test your way to product transformation, and Tony Fadell explains why in his new book Build:
“A/B and user testing is not product design. It’s a tool. A test. At best a diagnosis. It can tell you something’s not working, but it won’t tell you how to fix it. Or it can show you an option that solves one hyperlocal issue but breaks something else downstream.” — Tony Fadell.
That final point is so critical. A/B testing done poorly will result in runaway butterfly effects, where these small tests lead up to big, unknown changes long term in your products.
A/B testing is what you should do once you know the product design — once you have the vision. If you don’t know the vision and why you are building something, you shouldn’t be anywhere near A/B testing.
Fadell also talks about how user panels/focus groups aren’t very useful. You can’t ask users what to build (but you can ask them about their pain points). Why? Because they can’t conceive of what has never been built before.
I highly recommend Fadell’s new book Build to anyone looking to either learn how great products and experiences get made or to learn how to get better at product development. It’s one of the best product and business books I have read in years.