I was speaking on a panel the other day that was handed the topic, “the challenges of balancing data-light product bets vs purely data driven incremental improvements.” Camille Fournier was also a panelist and wrote up her thoughts here. Camille’s take (which I think is right) is that even if you don’t have data to work from, you can still approach projects analytically.
For me, the process of behaving analytically incorporates mathematical reasoning but not necessarily data. And I think this kind of spitballing is a useful activity, even if the numbers are made up. The reason for this is that human brains were forged on the African savanna where nothing is very fast, very large, or very small, cosmically speaking, and we are laughably equipped for coping with orders of magnitude.
The kind of thinking I’m describing works like this: “ok that’s a thing measured in thousands multiplied by a thing measured in tens of thousands, and then filtered through a rate of a few percent, are we even close?”. When permitted to skip this check on deficient intuition, most humans will sense their way to the wrong answers.
But on the panel and in subsequent discussions, it’s been easy to run with the dichotomy that you’ve either got data to work from, or you’ve got nothing at all. The temptation is to jump into philosophical takes given examples of products or entire markets that could not have been calculated with forsight before they existed. While that’s valid, I think it doesn’t describe most of the situations that you encounter in the wild.
Data Exists, and We Don’t Want to Look
The daily grind at a company consists of building in proximity to a thing that’s satisfying some definition of “working.” Yes, there’s always the innovator’s dilemma to worry about and the prospect of weird new platforms that will enable use cases you don’t understand yet. But the degree to which we’re striking out into the undiscovered country is overstated.
Companies release products that you’d figure shouldn’t have survived opportunity analysis all the time. They just don’t pitch them that way:
This feature notifies pairs of individuals that have arranged an unlikely relationship on the internet beforehand. The notifications are delivered two or three times a year, and only if the parties are in close geographic proximity. And they both have an optional iOS app installed. And in this scenario one of the people is known to be in a cohort that tends to not have that iOS app installed. And then at the end of this funnel we’re hoping that some small percentage of these folks will wind up showing up online and buying a thing. Later.
I have a real launch in mind with that, but I’ve rendered it unrecognizable and absurd by describing it accurately. This isn’t a situation where the volume couldn’t be estimated. If it were, I’d have a harder time lampooning it. This is the neglected scenario: we have all the data we need, but instead of deploying it we shipped something doomed.
When you hear people speak in defense of such things, they act out the same misdirection and head straight for the words we use when we’re discussing the iPod. You can’t, like, quantify vision, man. What they’re really espousing is the idea that product success obeys an uncertainty principle. If we look at things too closely, the magic disappears. And of course the good vibes would sublimate in this case, because the magic is nonsense.
The Hazards of Narrative Arc
Of course, this is not what anyone is actually thinking. Nobody sets out to ignore data on purpose, hoping to improve their chances of failing. You just watched me retcon an ethos onto feral behavior. And in doing so, I am part of the problem.
Everyone’s the hero of the novel they’re writing in their heads. That is the human condition. And having saved a company by inventing a new market is a great narrative arc, which is why we reach for it when we’re actually engaged in something mundane. We just systematically find stories too compelling.
It is rarely the case that vision can’t be at least sketched using arithmetic. Mathematics is the language we use to describe reality, and vision is generally assumed to have effects in reality. That’s what makes numeric methods more powerful than they should reasonably be. We’re constantly engaged in the art of self-deception, and they force you to snap out of it.