Dan McKinley

Some Recent Work

2017-05-08T00:00:00+00:00

Here are some links to recent work I’ve done elsewhere.

Ship Small Diffs - I tried to transmute the anguish I feel looking at huge changesets into words.
Mistakes You Apparently Just Have to Make Yourself - Getting youngfolk to listen to you is harder than I realized.
Fourteen Months with Clojure - Going back to my Lisp roots here.
The Push Train - Trying to frantically document some of the human element of making engineering function at a high level, which for whatever reason didn’t strike me as vital at the time.
Deploying Often is a Very Good Idea - Conditional probability is extremely good.
You Can’t Have a Rollback Button - Please engrave “but what if you didn’t?” on my tombstone I guess.
A Simple Pattern for Jobs and Crons on AWS - Not only did I stoop to writing a practical post for once, I also wrote a followup.
No Way Out but Through - More ranting and raving about deploying more than once a year.

The Unreasonable Effectiveness of Mathematics in Planning

2016-02-03T00:00:00+00:00

I was speaking on a panel the other day that was handed the topic, “the challenges of balancing data-light product bets vs purely data driven incremental improvements.” Camille Fournier was also a panelist and wrote up her thoughts here. Camille’s take (which I think is right) is that even if you don’t have data to work from, you can still approach projects analytically.

For me, the process of behaving analytically incorporates mathematical reasoning but not necessarily data. And I think this kind of spitballing is a useful activity, even if the numbers are made up. The reason for this is that human brains were forged on the African savanna where nothing is very fast, very large, or very small, cosmically speaking, and we are laughably equipped for coping with orders of magnitude.

That is also why you think this looks awesome, but don't let that spoil it for you.

The kind of thinking I’m describing works like this: “ok that’s a thing measured in thousands multiplied by a thing measured in tens of thousands, and then filtered through a rate of a few percent, are we even close?”. When permitted to skip this check on deficient intuition, most humans will sense their way to the wrong answers.

But on the panel and in subsequent discussions, it’s been easy to run with the dichotomy that you’ve either got data to work from, or you’ve got nothing at all. The temptation is to jump into philosophical takes given examples of products or entire markets that could not have been calculated with forsight before they existed. While that’s valid, I think it doesn’t describe most of the situations that you encounter in the wild.

Data Exists, and We Don’t Want to Look

The daily grind at a company consists of building in proximity to a thing that’s satisfying some definition of “working.” Yes, there’s always the innovator’s dilemma to worry about and the prospect of weird new platforms that will enable use cases you don’t understand yet. But the degree to which we’re striking out into the undiscovered country is overstated.

Companies release products that you’d figure shouldn’t have survived opportunity analysis all the time. They just don’t pitch them that way:

This feature notifies pairs of individuals that have arranged an unlikely relationship on the internet beforehand. The notifications are delivered two or three times a year, and only if the parties are in close geographic proximity. And they both have an optional iOS app installed. And in this scenario one of the people is known to be in a cohort that tends to not have that iOS app installed. And then at the end of this funnel we’re hoping that some small percentage of these folks will wind up showing up online and buying a thing. Later.

I have a real launch in mind with that, but I’ve rendered it unrecognizable and absurd by describing it accurately. This isn’t a situation where the volume couldn’t be estimated. If it were, I’d have a harder time lampooning it. This is the neglected scenario: we have all the data we need, but instead of deploying it we shipped something doomed.

When you hear people speak in defense of such things, they act out the same misdirection and head straight for the words we use when we’re discussing the iPod. You can’t, like, quantify vision, man. What they’re really espousing is the idea that product success obeys an uncertainty principle. If we look at things too closely, the magic disappears. And of course the good vibes would sublimate in this case, because the magic is nonsense.

The Hazards of Narrative Arc

Of course, this is not what anyone is actually thinking. Nobody sets out to ignore data on purpose, hoping to improve their chances of failing. You just watched me retcon an ethos onto feral behavior. And in doing so, I am part of the problem.

Everyone’s the hero of the novel they’re writing in their heads. That is the human condition. And having saved a company by inventing a new market is a great narrative arc, which is why we reach for it when we’re actually engaged in something mundane. We just systematically find stories too compelling.

It is rarely the case that vision can’t be at least sketched using arithmetic. Mathematics is the language we use to describe reality, and vision is generally assumed to have effects in reality. That’s what makes numeric methods more powerful than they should reasonably be. We’re constantly engaged in the art of self-deception, and they force you to snap out of it.

Do You Work at Amazon?

2016-01-26T00:00:00+00:00

Please note that Roberto Medri is a coauthor on this post.

Albert Wenger has been one of the VC’s I most admire for a long time. He was very present in the early days at Etsy, and sat in giving counsel on some, uh, significantly astray engineering team meetings. Albert is a smart, data-driven guy whose values roughly align with my own.

That said, I have an axe to grind with his latest post, Don’t Mind the Share Price. In it, Albert deploys the story of Amazon as a warning against focusing too much on how the market values a company. This is the story of Amazon:

Amazon was riding high in the late 90s, then felt the DotCom burst roughly along with the rest of the tech sector. Albert points out that history has shamed anyone that might’ve judged Amazon on its share price fifteen years ago, since it’s returned north of 2000% in the years since.

So whether you are running a tech company, working for one, or investing in one I highly recommend not reading too much into changes in share price. Focus instead on whether your company is making real progress.

Albert is careful to stress that you should focus on fundamentals over fluctuations in the price, which is generally good advice. But I think the subtext is clear: don’t be discouraged by even large declines in price, because you might be working at the next Amazon.

This is a premise that we can investigate quantitatively.

The Odds of Being an Amazon

Suppose that we’re working at a public company that’s experienced a decline in its share price of at least 50%, relative to a recent high price. We’d like to approximate the odds that this company is going to recover [1].

It turns out that since 2002, there have been 2,132 companies traded on the NASDAQ that fit this description. One of these is indeed Amazon. But how many others are like it?

We can take this set of companies and categorize them. Let’s identify companies that wound up being completely wiped out—losing 90% of their remaining value or more—and then all other companies that declined in value. For companies that increased in value, we’ll differentiate those that beat the market (defined as the S&P 500 Index) from those that didn’t. The idea being that you would have been better off just buying an index fund with your cash surplus from working for Google in a parallel universe. And finally, we’ll identify the miraculous: those companies that return 1000% or more, of which Amazon is one example.

If we do that, it looks like this:

Category	Count	Percent	Cumulative Percent
Wiped Out	239	11.21%	11.21%
Declined	794	37.24%	48.45%
Beaten by Market	344	16.14%	64.59%
Beat Market	661	31.00%	95.59%
Miracle	94	4.41%	100.00%

Here we can see that about 65% percent of public companies that find themselves in this situation don’t recover. But 35% of companies do. These are tough odds, but definitely not impossible odds, right?

Recovery is not Good Enough

Albert asks us to consider investors, officers, and employees of the company as having roughly identical situations. This is a mistake. Things are significantly worse in the case of employees [2] at a public company that have been issued options. In these cases, the company may very well recover, but we have to contemplate several other horrifying possibilities.

Employees may have already exercised options at a strike price higher than the current market price. If so, they’re screwed if the company never recovers above that price. Even if the company beats the market from here out.
The strike price may be below the current market price, meaning that the options are worth something. But employees may owe taxes (or AMT), forcing them to sell before the recovery.
Options may be underwater and worthless. At least in this scenario, there is clarity.

From these situations we can see that as an employee [3], it makes sense to consider the odds that the company will not just recover, but will ultimately get back to where it was. That looks like this:

Category	Count	Percent	Cumulative Percent
Wiped Out	239	11.21%	11.21%
Declined	794	37.24%	48.45%
Beaten by Market	344	16.14%	64.59%
Recovered Below High Price	210	9.84%	75.04%
Beat Market	441	20.66%	95.69%
Miracle	92	4.31%	100.00%

This makes it worse: 75% of companies won’t recover using this definition. And only about 4% will make miraculous comebacks of Amazon’s order of magnitude.

Are You Making Progress?

Remember that Albert provides us with an important caveat: we should “[f]ocus … on whether the company is making real progress.” But this can be tricky to surmise as an employee, for several reasons:

You are in unavoidably close proximity to a coordinated propaganda campaign. It’s called the company’s internal communications and morale efforts. You may find yourself thinking unreasonably positively about these things.
You are putting in hours at this company, and human nature compels us to confuse effort with progress.
Remember that we’re talking about a public company. So unless you’re an officer, you’ll have a difficult time of getting detailed information about how much progress the company is really making. And of course timing trades on such information would be illegal.

We should agree that the outlook here is going to be hazy at best, and self-deception is a hazard.

The Base Rate Fallacy's Perverse Tyranny Over the American [4] Mind

If there is any line of reasoning that really drives me crazy, it’s the following:

A series of cosmically unlikely events has unfolded.
This is submitted as evidence that it can happen to anyone.

Examples of this are everywhere. Someone is going to win powerball, therefore it makes sense to buy tickets. Barack Obama was elected president, therefore systematic racism is toothless. Mark Zuckerberg struck it rich, so you’ve just gotta have faith.

By the way this guy also thinks that picking your own numbers gives you a higher chance of winning.

In looking to Amazon (or Google, Facebook, Netflix, or dear god Apple) as consolation in the event that a company has experienced a decline in share price, we make the following mistake. The probability that successful companies have stumbled in their past is not the probability that a company will succeed, having stumbled.

This isn’t a call for nihilism if you find yourself in such a situation. Far from it—it’s a call to realize that the odds are now against you, and to behave proactively.

The code and data for this article is available here, on Github. It’s a bit sloppy and hastily written, sorry. We started from a dataset of companies traded on the NASDAQ that experienced a decline of 50% or more off of a previous high. our dataset started around the year 2000.

You may notice that I've switched questions, from "are you working at Amazon" to "is the company Amazon." Calculating the odds that you are working at Amazon would of course require a richer dataset that includes company headcounts, and I am a lazy man.
Investors can more easily scale their commitment to the company by having a diverse portfolio. Employees and officers, however, give 100% of their labor to the company. And in the event that things go well, a large percentage of their net worth derives from the value of the company. Officers have a high floor on their returns, via guaranteed bonuses, parachute provisions, accelerated vesting schedules in the event of termination, and so on. Employees on the other hand are screwed.
This refinement doesn't apply to all employees. Early employees probably have strike prices that are very low, and can make money despite a large drop in the share price. But at a newly-minted public company, most employees are probably new, and most employees are therefore affected.
I know that Albert Wenger is German.

Are My Push Notifications Driving Users Away?

2015-11-24T00:00:00+00:00

In response to Kellan’s musing about push notifications on twitter, Adam McCue asked an interesting question:

@kellan @mcfunley what's the best way to do this?
— Adam McCue (@mccue) November 25, 2015

I quickly realized that fitting an answer into tweets was hopeless, so here’s a stab at it in longform.

How would we do this?

Let’s come up with a really simple way to figure this out for the case of a single irritating notification. This is limited, but the procedure described ought to be possible for anyone with a web-enabled mobile app. We need:

A way to divide the user population into two groups: a treatment group that will see the ad notification, and a control group that won’t.
A way to decide if users have disappeared or not.

To make the stats as simple as possible, we need (1) to be random and we need (2) to be a binomial measure (i.e. “yes or no,” “true or false,” “heads or tails,” etc).

To do valid (simple) stats, we also want our trials to be independent of each other. If we send the same users the notifications over and over, we can’t consider each of those to be independent trials. It’s easy to intuit why that might be: I’m more likely to uninstall your app after the fifth time you’ve bugged me [1]. So we need to consider disjoint sets of users on every day of the experiment.

Does this hurt us or help us? Let's try science.

How to randomly select users to receive the treatment under these conditions is up to you, but one simple way that should be broadly applicable is just hashing the user ID. Say we need 100 groups of users: both a treatment and control group for 50 days. We can hash the space of all user ID’s down to 100 buckets [2].

So how do we decide if users have disappeared? Well, most mobile apps make http requests to a server somewhere. Let’s say that we’ll consider a user to be “bounced” if they don’t make a request to us again within some interval.

Some people will probably look at the notification we sent (resulting in a request or two), but be annoyed and subsequently uninstall. We wouldn’t want to count such a user as happy. So let’s say we’ll look for usage between one day after the notification and six days after the notification. Users that send us a request during that interval will be considered “retained.”

Some examples of our binomial model. We'll call a user retained if they request data from us on any of days two through seven counting from the time of the notification. User 4 in this example is not retained because (s)he only requests data on the day the notification was sent.

To run the experiment properly you need to know how long to run it. That depends a lot on your personal details: how many people use your app, how often they use it, how valuable the ad notification is, and how severe uninstalls are for you. For the sake of argument, let’s say:

We can find disjoint sets of 10,000 users making requests to us on any given day, daily, for a long time.
(As discussed) we’ll put 50% of them in the treatment group.
60% of people active on a given day currently will be active between one and six days after that.
We want to be 80% sure that if we move that figure by plus or minus 1%, we’ll know about it.
We want to be 95% sure that if we measure a deviation in plus or minus 1% that it’s for real.

If you plug all of that into experiment calculator [3] it will tell you that you need 21 days of data to satisfy those conditions. But since we use a trailing time interval in our measurement, we need to wait 28 days.

An example result

Ok, so let’s say we’ve run that experiment and we have some results. And suppose that they look like this:

Group	Users	Retained users	Bounced users
Treatment	210,000	110,144	99,856
Control	210,000	126,033	83,967

Using these figures we can see that we’ve apparently decreased retention by 12.6%, and a test of proportions confirms that this difference is statistically significant. Oops!

I’ve run the experiment, now what?

You most likely have created the ad notification because you had some positive goal in mind. Maybe the intent was to get people to buy something. If that’s the case, then you should do an additional computation to see if what you gained in positive engagement outweighs what you’ve lost in users.

I don’t think I have enough data.

You might not have 420,000 users to play with, but that doesn’t mean that the experiment is necessarily pointless. In our example we were trying to detect changes of plus or minus one percent. You can detect more dramatic changes in behavior with smaller sets of users. Good luck!

I’m sending reactivation notifications to inactive users. Can I still measure uninstalls?

In our thought experiment, we took it as a given that users were likely to use your app. Then we considered the effect of push notifications on that behavior. But one reason you might be contemplating sending the notifications is that they’re not using it, and you are trying to reactivate them.

If that’s the case, you might want to just measure reactivations instead. After all, the difference between a user who has your app installed but never opens it and a user that has uninstalled your app is mostly philosophical. But you may also be able to design an experiment to detect uninstalls. And that might be sensible if very, very infrequent use of your app can still be valuable.

A procedure that might work for you here is to send two notifications. You could then use delivery failures of secondary notifications as a proxy metric for uninstalls.

I want to learn more about this stuff.

As it happens, I recorded a video with O’Reilly that covers things like this in more detail. You might also like Evan Miller’s blog and Ron Kohavi’s publications.

"How many notifications are too many?" is a separate question, not considered here.
If you do many experiments, you want to avoid using the _same_ sets of people as control and treatment. So include something based on the name of the experiment in the hash. So if user 12345 is in the treatment for 50/50 experiment X, she should be only 50% likely (not 100% likely) to be in the treatment for some other 50/50 experiment Y.
The labeling on the tool is for experiments on a website. The math is the same though.

Choose Boring Technology (Expanded, Slide-Based Edition)

2015-07-27T00:00:00+00:00

I gave a spoken word version of Choose Boring Technology at OSCON in Portland last week. Here are the slides:

Choose Boring Technology

2015-03-30T00:00:00+00:00

Probably the single best thing to happen to me in my career was having had Kellan placed in charge of me. I stuck around long enough to see Kellan’s technical decisionmaking start to bear fruit. I learned a great deal from this, but I also learned a great deal as a result of this. I would not have been free to become the engineer that wrote Data Driven Products Now! if Kellan had not been there to so thoroughly stick the landing on technology choices.

Being inspirational as always.

In the year since leaving Etsy, I’ve resurrected my ability to care about technology. And my thoughts have crystallized to the point where I can write them down coherently. What follows is a distillation of the Kellan gestalt, which will hopefully serve to horrify him only slightly.

Embrace Boredom.

Let’s say every company gets about three innovation tokens. You can spend these however you want, but the supply is fixed for a long while. You might get a few more after you achieve a certain level of stability and maturity, but the general tendency is to overestimate the contents of your wallet. Clearly this model is approximate, but I think it helps.

If you choose to write your website in NodeJS, you just spent one of your innovation tokens. If you choose to use MongoDB, you just spent one of your innovation tokens. If you choose to use service discovery tech that’s existed for a year or less, you just spent one of your innovation tokens. If you choose to write your own database, oh god, you’re in trouble.

Any of those choices might be sensible if you’re a javascript consultancy, or a database company. But you’re probably not. You’re probably working for a company that is at least ostensibly rethinking global commerce or reinventing payments on the web or pursuing some other suitably epic mission. In that context, devoting any of your limited attention to innovating ssh is an excellent way to fail. Or at best, delay success [1].

What counts as boring? That’s a little tricky. “Boring” should not be conflated with “bad.” There is technology out there that is both boring and bad [2]. You should not use any of that. But there are many choices of technology that are boring and good, or at least good enough. MySQL is boring. Postgres is boring. PHP is boring. Python is boring. Memcached is boring. Squid is boring. Cron is boring.

The nice thing about boringness (so constrained) is that the capabilities of these things are well understood. But more importantly, their failure modes are well understood. Anyone who knows me well will understand that it’s only with a overwhelming sense of malaise that I now invoke the spectre of Don Rumsfeld, but I must.

To be clear, fuck this guy.

When choosing technology, you have both known unknowns and unknown unknowns [3].

A known unknown is something like: we don’t know what happens when this database hits 100% CPU.
An unknown unknown is something like: geez it didn’t even occur to us that writing stats would cause GC pauses.

Both sets are typically non-empty, even for tech that’s existed for decades. But for shiny new technology the magnitude of unknown unknowns is significantly larger, and this is important.

Optimize Globally.

I unapologetically think a bias in favor of boring technology is a good thing, but it’s not the only factor that needs to be considered. Technology choices don’t happen in isolation. They have a scope that touches your entire team, organization, and the system that emerges from the sum total of your choices.

Adding technology to your company comes with a cost. As an abstract statement this is obvious: if we’re already using Ruby, adding Python to the mix doesn’t feel sensible because the resulting complexity would outweigh Python’s marginal utility. But somehow when we’re talking about Python and Scala or MySQL and Redis people lose their minds, discard all constraints, and start raving about using the best tool for the job.

Your function in a nutshell is to map business problems onto a solution space that involves choices of software. If the choices of software were truly without baggage, you could indeed pick a whole mess of locally-the-best tools for your assortment of problems.

The way you might choose technology in a world where choices are cheap: "pick the right tool for the job."

But of course, the baggage exists. We call the baggage “operations” and to a lesser extent “cognitive overhead.” You have to monitor the thing. You have to figure out unit tests. You need to know the first thing about it to hack on it. You need an init script. I could go on for days here, and all of this adds up fast.

The way you choose technology in the world where operations are a serious concern (i.e., "reality").

The problem with “best tool for the job” thinking is that it takes a myopic view of the words “best” and “job.” Your job is keeping the company in business, god damn it. And the “best” tool is the one that occupies the “least worst” position for as many of your problems as possible.

It is basically always the case that the long-term costs of keeping a system working reliably vastly exceed any inconveniences you encounter while building it. Mature and productive developers understand this.

Choose New Technology, Sometimes.

Taking this reasoning to its reductio ad absurdum would mean picking Java, and then trying to implement a website without using anything else at all. And that would be crazy. You need some means to add things to your toolbox.

An important first step is to acknowledge that this is a process, and a conversation. New tech eventually has company-wide effects, so adding tech is a decision that requires company-wide visibility. Your organizational specifics may force the conversation, or they may facilitate developers adding new databases and queues without talking to anyone. One way or another you have to set cultural expectations that this is something we all talk about.

One of the most worthwhile exercises I recommend here is to consider how you would solve your immediate problem without adding anything new. First, posing this question should detect the situation where the “problem” is that someone really wants to use the technology. If that is the case, you should immediately abort.

I just watched a webinar about this graph database, we should try it out.

It can be amazing how far a small set of technology choices can go. The answer to this question in practice is almost never “we can’t do it,” it’s usually just somewhere on the spectrum of “well, we could do it, but it would be too hard” [4]. If you think you can’t accomplish your goals with what you’ve got now, you are probably just not thinking creatively enough.

It’s helpful to write down exactly what it is about the current stack that makes solving the problem prohibitively expensive and difficult. This is related to the previous exercise, but it’s subtly different.

New technology choices might be purely additive (for example: “we don’t have caching yet, so let’s add memcached”). But they might also overlap or replace things you are already using. If that’s the case, you should set clear expectations about migrating old functionality to the new system. The policy should typically be “we’re committed to migrating,” with a proposed timeline. The intention of this step is to keep wreckage at manageable levels, and to avoid proliferating locally-optimal solutions.

This process is not daunting, and it’s not much of a hassle. It’s a handful of questions to fill out as homework, followed by a meeting to talk about it. I think that if a new technology (or a new service to be created on your infrastructure) can pass through this gauntlet unscathed, adding it is fine.

Just Ship.

Polyglot programming is sold with the promise that letting developers choose their own tools with complete freedom will make them more effective at solving problems. This is a naive definition of the problems at best, and motivated reasoning at worst. The weight of day-to-day operational toil this creates crushes you to death.

Mindful choice of technology gives engineering minds real freedom: the freedom to contemplate bigger questions. Technology for its own sake is snake oil.

Update, July 27th 2015: I’ve produced a talk based on this article. You can see it here.

Etsy in its early years suffered from this pretty badly. We hired a bunch of Python programmers and decided that we needed to find something for them to do in Python, and the only thing that came to mind was creating a pointless middle layer that [required years of effort to amputate](https://www.youtube.com/watch?v=eenrfm50mXw). Meanwhile, the 90th percentile search latency was about two minutes. [Etsy didn't fail](http://www.sec.gov/Archives/edgar/data/1370637/000119312515077045/d806992ds1.htm), but it went several years without shipping anything at all. So it took longer to succeed than it needed to.
We often casually refer to the boring/bad intersection of doom as "enterprise software," but that terminology may be imprecise.
In saying this Rumsfeld was either intentionally or unintentionally alluding to [the Socratic Paradox](http://en.wikipedia.org/wiki/I_know_that_I_know_nothing). Socrates was by all accounts a thoughtful individual in a number of ways that Rumsfeld is not.
A good example of this from my experience is [Etsy's activity feeds](https://speakerdeck.com/mcfunley/etsy-activity-feed-architecture). When we built this feature, we were working pretty hard to consolidate most of Etsy onto PHP, MySQL, Memcached, and Gearman (a PHP job server). It was much more complicated to implement the feature on that stack than it might have been with something like Redis (or [maybe not](https://aphyr.com/posts/283-call-me-maybe-redis)). But it is absolutely possible to build activity feeds on that stack.

An amazing thing happened with that project: our attention turned elsewhere for several years. During that time, activity feeds scaled up 20x while nobody was watching it at all. We made no changes whatsoever specifically targeted at activity feeds, but everything worked out fine as usage exploded because we were using a shared platform. This is the long-term benefit of restraint in technology choices in a nutshell.

This isn't an absolutist position--while activity feeds stored in memcached was judged to be practical, implementing full text search with faceting in raw PHP wasn't. So Etsy used Solr.

Data Driven Products: Lean Startup 2014

2015-01-27T00:00:00+00:00

Here’s a video of me doing a slightly-amended version of my Data Driven Products talk at the Lean Startup Conference back in December.

I am told I upspeak? You be the judge.

Thoughts on the Technical Track

2014-12-09T00:00:00+00:00

I saw lizTheDeveloper’s post about technical leadership at Simple and I realized that I’ve been meaning to write about this for a while. I hope to persuade you that there are a number of systemic biases working against a healthy technical career path. I don’t think that they’re insurmountable, and I don’t disagree with Liz’s post. But I’ve never heard of a company clearing all of these hurdles at once.

I was the first person at Etsy with the title of “Principal Engineer,” which was the technical equivalent to a directorship (i.e., one level below CTO). I’m not saying this to toot my own horn, but rather so that it’s understood that the following comes from someone that was the beneficiary of an existing system.

(Incidentally, I think Etsy is an example of a company whose heart is in the right place, and it’s not my intention to single them out.)

To Review, Management is a Job

My views on the merits of having a technical track align with those of many people in our industry. Management is a different job, with different skills. They’re not necessarily more difficult skills, they’re just different. By and large they’re unrelated to the day-to-day labor of the people who build technology products.

It doesn’t make any sense to divert your technical talent into a discipline where they will need to stop doing technical work. (That’s in the event that they intend to be effective managers, which I concede might be an unrealistic expectation.)

Other people have made this case, so I’ll just proceed as if we agree that there must be a way forward for people that are great programmers other than to simply graduate into not programming at all.

Having that way forward is an ideal. There is always a gap between our ideals and reality, and we cannot act as though we’ve solved a problem simply by articulating it.

Fundamental Asymmetries

Management Just Happens

I have had management responsibility thrust upon me at least four times over the course of my career, and at no point has that been my goal. It just happens. Do you want to be a manager? I will now tell you the secret to becoming a manager in a growing company: just wait.

You have a manager. Eventually, your manager will accrue too many responsibilities, and they will freak out. They will need somebody to take over some of their reports, and that lucky warm body is you.

Good hair: also helpful.

It is entirely plausible to become a manager accidentally. It might even be the norm.

Technical Track Promotions are Post-Hoc

The process for minting a new manager is: crap, we need another manager. There’s no symmetrical forcing function pushing people into the upper ranks of technical leadership.

Mentorship and technical feedback are things everyone does on a functioning engineering team. A technical track “promotion” is merely additional recognition given to someone who is already performing that role notably well.

If the job is already getting done, then filling the job is clearly not a pressing need. Technical promotions are something that happen when it’s convenient, which is generally never.

Stumping

Between the founding of the United States and the end of the 19th century, it was considered tacky for presidential candidates to personally campaign for the job. Instead, they staged an elaborate farce in which they reluctantly answered the call of the nation to serve. Trying to intentionally get a promotion into the technical track is pretty much just like this.

Getting promoted in the technical track is kind of like being James Garfield.

Your work must be recognized, and this is the rub. Let me rephrase: “someone with the power to bestow promotions has to be your fan.” To be promoted you have to be a good mentor, but you also have to worry about playing to an audience. That may be executives, or it may be your peers (and potential competitors). Regardless, you’re running a weird campaign in which actually saying anything directly about wanting the job would be gauche.

The most qualified individual contributors may become known without ever really doing this on purpose, but that doesn’t say much for this as a tenable career goal of the sort that can be counted on.

The Problem of Credibility

Society Applies to Idealistic Tech Companies, Too

American society is not a classless oasis. That’s a lie we tell ourselves. And the person who knows what everyone else gets paid and can fire you is not in your class.

A technical job does not have equivalent prestige to a management position with an equivalent salary just because you say it does. Even if you conquer this within your own company, it’s not true in the rest of the industry, and it’s not true in the world at large. In the world our parents live in, it’s a big deal to be somebody else’s boss.

You’re hiring people from the world at large all the time. Without continuous effort a technical track decays to its ground state, where the jobs are second class.

Halfhearted Managers are The Worst

The natural result of a system in which technical promotions can’t be counted on and are viewed as suspiciously-maybe-second-class anyway is that people who don’t really give a shit about management wind up going into management. Given the choice of waiting for a technical promotion that may never arrive and taking an offer to manage others, almost everyone is going to take the bird in the hand.

Once you let the soulless suspendered lizard in the building, you are screwed.

Managers that have no passion for management are a blight on society. I can say this because I have been one of them. I was never a good manager, and for that I apologize to anyone that ever had to report to me.

I am not an isolated case. Many people in management are frankly terrible at it. And they would rather have technical track jobs anyway, but they have no idea how to make the switch. A credible technical track is a great way to ensure a higher level of satisfaction and competency among the managers.

Ratios Observed in the Wild Make No Sense

You don’t need to take my reasoning about the intrinsic pressure favoring management bloat at face value. You can actually look at the ratio of managers to technical employees at your company.

At one point, I was alone at my level. There were five theoretically-equivalent directors at the time. The ratio was at least that bad on the lower rungs. (I have no idea if this is still true at that company, and it might not be.)

For that to make sense, we’d have to believe a few things that don’t stand up to scrutiny. First, we’d have to believe in a very high proclivity among engineers to manage, and I think that betrays our expectations. Not very many of us got into this business with the hope of not actually building things.

Second, we’d have to believe that although it took five directors to effectively manage the organization, only one technical leader was required to advise the same group on the details of the work they do every day.

What Might Help?

Promotions Should Not Be Miraculous and Rare

Of course, it wouldn’t make logical sense to say that the ratio of individual contributors to managers at a given level must be 1:1. I honestly don’t know if 1:2 or 2:1 is closer to correct. The answer is probably contingent, and the relationship might not be linear.

But I think it’s important for any company that takes the ideal of having a tenable technical track seriously to put a stake in the ground on this question. It’s hard to build a credible technical track, and we need a baseline to grade ourselves against.

I don’t think that proceeding with the assumption that leaders will just naturally emerge produces the best results. Adding a self-imposed quota achieves accountability. It acknowledges the possibility that problems can lie in the system of recognition, and not only in the talents of the people in the pool for promotions.

“Do we think that we hire smart people here? Yes? Then we should be able to find N of them worthy of promotion for every manager. If we can’t then the problem is most likely to be found in how we’re recognizing people for their work.”

I know that the word “quota” is verboten for many, and I gleefully await your flames.

Address Prestige with Superpowers

If we think about why managers and technical employees on even salary footing may be perceived to not truly be equals, it comes down to superpowers. The managers have special capabilities that the technical employees don’t: hiring, firing, compensation, and the like. Is it possible to give technical employees a different set of superpowers, to address the prestige problem?

Maybe. I don’t think that I have seen this done correctly yet. If I had superpowers, they were:

The ability to work on whatever I wanted.
The ability to talk to anyone I wanted.

These were indeed powerful, but using them to create positive action was difficult. It would have been easy for me to opt out of projects that I didn’t believe in and to do my own thing. I did often do my own thing. But I also worked on projects that I didn’t believe in, because I knew that opting out was a selfish act. One of my friends would just be forced to work on it in my place, and sometimes leadership is about jumping on grenades.

I guess there are worse superpowers. For example, the ability to allow oneself to be framed for the good of the city.

Talking to other teams made it possible for me to point out places where resources weren’t intelligently allocated. But this also begat mostly negative actions. “Hey, this isn’t the best way to use these folks,” I’d find myself saying all the time. It was draining, and a bummer.

Giving the technical leadership deeper involvement in the planning process could address this. Of course that would involve dragging the technical leadership to meetings, which I admit is tricky.

In Closing

I hope I’ve demonstrated that creating a career path outside of management for technical employees is only the beginning of your problems. It’s a good and necessary step, but it’s not an achievement by itself.

I’d love to hear from anyone with better ideas. These issues are difficult and I don’t claim to have all of the right answers.

Data Driven Products Now!

2014-09-18T00:00:00+00:00

Back when I was at Etsy, I did a presentation internally about the craft of sizing opportunities. I finally got around to writing a public incarnation of that talk. Here it is:

Manual Delivery

2014-03-10T00:00:00+00:00

The person on build rotation, or the nightly schlimazel I suppose, went into a hot 5’x8’ closet containing an ancient computer. This happened after everyone else had left, so around 8:30PM. Although in crunch time that was more like 11:30PM. And we were in crunch time at one point for a stretch of a year and a half. “That release left a mark,” my friend Matt used to say. In a halfhearted attempt at fairness to those who will take this post as a grave insult, I’ll concede that my remembrance of these details is the work of The Mark.

Anyway, the build happened after quitting time. This guaranteed that if anything went wrong, you were on your own. Failure in giving birth to the test build implied that the 20 people in Gurgaon comprising the QA department would show up for work in a matter of hours having nothing to do.

You used a tool called “VBBuild.” This was a GUI tool, rumored to be written by Russians:

VBBuild did mysterious COM stuff to create the DLLs that nobody at the time understood properly. It presented you with dozens of popups even when it was working perfectly, and you had to be present to dismiss each of them. The production of executable binary code was all smoke and lasers. And, apparently, popups.

Developers wrote code using the more familiar VB6 IDE. The IDE could run interpreted code as an interactive debugger, but it could not produce finished libraries in a particularly repeatable or practical way. So the release compilation was different in many respects from what programmers were doing at their desks. Were there problems that existed in one of these environments but not the other? Yes, sometimes. I recall that we had a single function that weighed in at around 70,000 lines. The IDE would give up and execute this function even if it contained clear syntax errors. That was the kind of discovery which, while exciting, was wasted in solitude somewhere past midnight as you attempted to lex and parse the code for keeps.

Isaiah 2:4: "And he shall displace VB6 in search engine results with a book written by vegans."

Developers weren’t really in the habit of doing complete pulls from source control. And who could blame them, since doing this whitescreened your machine for half an hour. They were also never in any particular hurry to commit, at least until it was time to do the test build. As there was no continuous integration at the time, this was the first time that all of the code was compiled in several days.

Often [ed: always] there were compilation errors to be resolved. We were using Visual Sourcesafe, so people could be holding an exclusive lock on files containing the errors. Typically, this problem was addressed by walking around the office an hour before build time and reminding everyone to check their files in. In the event that someone forgot [ed: every time], there was an administrative process for unlocking locked files. Not everyone had the necessary rights to do this, but happily, I did.

By design, the build tried to assume an exclusive lock on all of the code. As a result, nobody could work while the build was in progress. Sometimes, the person performing the build would check all of the files out and not check them back in. So your first act the morning after a build might be to walk over to the build closet and release the source files from their chains.

The Visual Sourcesafe documentation strongly advised against its use on a team of more than four programmers, and apparently this was not a joke.

Deployment required dozens of manual steps that I will never be able to remember. When the build was done, you copied DLLs over to the test machines and registered them there. By “copied” I mean that you selected them in an explorer window, pressed “Ctrl-C,” and then pressed “Ctrl-V” to paste them into another. There was no batch script worked out to do this more efficiently. Ok, this is a slight lie. There had been a script, but was put out to pasture on account of a history of hideous malfunction. And popups. On remote machines sometimes, where they could only be dismissed by wind and ghosts.

Registration involved connecting to each machine with Remote Desktop and right clicking all the DLLs. You could skip a machine or just one library, and things would be very screwy indeed.

The production release, which happened roughly twice a year under ideal conditions, was identical to this but with the added complexity of about eight more servers receiving the build. And we might take the opportunity to add completely new machines, which would not necessarily have the same patch levels for, oh, like 700,000 windows components that were relied upon.

Given eight or ten machines, the probability of a mistake on at least one of the servers approached unity. So the days and weeks following a production release were generally spent sussing out all of the minute differences and misconfigurations on the production machines. There would be catastrophic bugs that affected a tiny sliver of requests, under highly specific server conditions, and only if executed on one server out of eight. I was an expert at debugging in disassembly at the time. Upon leaving the job, I thought that this was pretty badass. But in the seven years since–do you know what? It’s never come up.

"The code could be structured by cows and we would build it by hand."

At one point I wrote a new script to perform the deployment. It was an abomination of XML to be sure, but it got the job done without all of the popups. I started doing the test build with this with some success and suggested that we use it for the production release. This was out of the question, I was told by one of my closer allies in the place. The production release was “too important to use a script.”

The operating systems and supporting libraries on the machines were also set up by hand, by a separate team, working from printed notes. The results were similar. This is kind of another story.

This all happened in 2003.

Scalding at Etsy

2014-03-02T00:00:00+00:00

Here’s a presentation I gave about how Etsy wound up using Scalding for analysis. Given at the San Francisco Cascading Meetup.

The Case for Secrecy in Web Experiments

2014-01-16T00:00:00+00:00

For four months ending in early 2011, I worked on team of six to redesign Etsy’s homepage. I don’t want to overstate the weight of this in the grand scheme of things, but hopes flew high. The new version was to look something like this:

There were a number of methodological problems with this, one of our very first web experiments. Our statistics muscles were out of practice, and we had a very difficult time fighting the forces of darkness who wanted to enact radical redesigns after five minutes of real-time data. We had no toolchain for running experiments to speak of. The nascent analytics pipeline jobs failed every single night.

But perhaps worst of all, we publicized the experiment. Well, “publicized” does not accurately convey the magnitude of what we did. We allowed visitors to join the treatment group using a magic URL. We proactively told our most engaged users about this. We tweeted the magic URL from the @Etsy account, which at that point had well over a million followers.

The magic URL was chosen to celebrate the CEO's 31st birthday. None of this was Juliet's fault.

This project was a disaster for many reasons. Nearly all of the core hypotheses turned out to be completely wrong. The work was thrown out as a total loss. Everyone involved learned valuable life lessons. I am here today to elaborate on one of these: telling users about the experiment as it was running was a big mistake.

The Diamond-Forging Pressure to Disclose Experiments

If you operate a website with an active community, and you do A/B testing, you might feel some pressure to disclose your work. And this seems like a proper thing to do, if your users are invested in your site in any serious way. They may notice anyway, and the most common reaction to change on a beloved site tends to be varying degrees of panic.

"If you can't beat 'em, join 'em."

As an honest administrator, your wish is to reassure your community that you have their best interest at heart. Transparency is the best policy!

Except in this case. I think there’s a strong argument to be made against announcing the details of active experiments. It turns out to be easier for motivated users to overturn your experiment than you may believe. And disclosing experiments is work, and work that comes before real data should be minimized.

Online Protests: Not Necessarily A Waste of Time

A fundamental reason that you should not publicize your A/B tests is that this can introduce bias that can affect your measurements. This can even overturn your results. There are many different ways for this to play out.

Most directly, motivated users can just perform positive actions on the site if they believe that they are in their preferred experiment bucket. Even if the control and treatment groups are very large, the number of people completing a goal metric (such as purchasing) may be just a fraction of that. And the anticipated difference between any two treatments might be slight. It’s not hard to imagine how a small group of people could determine an outcome if they knew exactly what to do.

Group	Visits	Conversions (organic)	Conversions (gamed)	Proportion
Control	10000	50	10	0.0060
New	10000	55	0	0.0055

Control	New
10000 visits	10000 visits
50 organic conversions	50 organic conversions
10 gamed conversions	0 gamed conversions
0.60% converted	0.55% converted

Figure 1: In some cases a small group of motivated users can change an outcome, even if the sample sizes are large.

As the scope and details of an experiment become more fully understood, this gets easier to accomplish. But intentional, organized action is not the only possible source of bias.

Even if users have no preference as to which version of a feature wins, some will still be curious. If you announce an experiment, visitors will engage with the feature immediately who otherwise would have stayed away. This well-intentioned interest could ironically make a winning feature appear to be a loss. Here’s an illustration of what that looks like.

Group	Visits (oblivious)	Visits (rubbernecking)	Visits (total)	Conversions	Proportion
Control	500	50	550	30	0.055
New	500	250	750	35	0.047

Control	New
500 oblivious visits	500 oblivious visits
50 rubbernecking visits	250 rubbernecking visits
550 total visits	750 total visits
30 conversions	35 conversions
5.5% converted	4.7% converted

Figure 2: An example in which 100 engaged users are told about a new experiment. They are all curious and seek out the feature. Those seeing the new treatment visit the new feature more often just to look at it, skewing measurement.

These examples both involve the distortion of numbers on one side of an experiment, but many other scenarios are possible. Users may change their behavior in either group for no reason other than that they believe they are being measured.

Good experimental practice requires that you isolate the intended change as the sole variable being tested. To accomplish this, you randomly assign visitors the new treatment or the old, controlling for all other factors. Informing visitors that they’re part of an experiment places this central assumption in considerable jeopardy.

Predicting Bias is Hard

“But,” you might say, “most users aren’t paying attention to our communiqués.” You may think that you can announce experiments, and only a small group of the most engaged people will notice. This is very likely true. But as I have already shown, the behavior of a small group cannot be dismissed out of hand.

Obviously, this varies. There are experiments in which a vocal minority cannot possibly bias results. But determining if this is true for any given experiment in advance is a difficult task. There is roughly one way for an experiment to be conducted correctly, and there are an infinite number of ways for it to be screwed.

A/B tests are already complicated: bucketing, data collection, experimental design, experimental power, and analysis are all vulnerable to mistakes. From this point of view, “is it safe to talk about this?” is just another brittle moving part.

Communication Plans are Real Work

Something I have come to appreciate over the years is the role of product marketing. I have been involved in many releases for which the act of explaining and gaining acceptance for a new feature constituted the majority of the effort. Launches involve a lot more than pressing a deploy button. This is a big deal.

Product marketing: this is serious business.

It also seems to be true that people who are skilled at this kind of work are hard to come by. You will be lucky to have a few of them, and this imposes limits on the number of major changes that you can make in any given year.

It makes excellent sense to avoid wasting this resource on quite-possibly-fleeting experiments. It will delay their deployment, steal cycles from launches for finished features, and it will do these things in the service of work that may never see the light of day!

Users will tend to view any experiment as presaging an imminent release, regardless of your intentions. Therefore, you will need to put together a relatively complete narrative explaining why the changes are positive at the outset. A “minimum viable announcement” probably won’t do. And you will need to execute this without the benefit of quantitative results to bolster your case.

Your Daily Reminder that Experiments Fail

Doing data-driven product work really does imply that you will not release changes that don’t meet some quantitative standard. In such an event you might tweak things and start over, or you might give up altogether. Announcing your running experiments is problematic given this reality.

Obviously, product costs will be compounded by communication costs. Every time you retool an experiment, you will have to bear the additional weight of updating your community. Adding marginal effort makes it more difficult for humans to behave rationally and objectively. We have a name for this well-known pathology: the sunk cost fallacy. We’ve put so much into this feature, we can’t just give up on it now.

The fear of admitting mistakes in public can be motivating.

Announcing experiments also has a way of raising the stakes. The prospect of backtracking with your users (and being perceived as admitting a mistake) only makes killing a bad feature less palatable. The last thing you need is additional temptation to delude yourself. You have plenty of this already. The danger of living in public is that it will turn a bad release that should be discarded into an inevitability.

Consistency and Expectations

Let’s say you’ve figured out workarounds for every issue I’ve raised so far. You are still going to want to run experiments that are not publicly declared.

Some experiments are inherently controversial or exploratory. It may be perfectly legitimate to try changes that you would never release to learn more about your site. Removing a dearly beloved feature temporarily for half of new registrations is a good example of this. By doing so, you can measure the effect of that feature on lifetime value, and make better decisions with your marketing budget.

Other experiments work only when they’re difficult to detect. Search ranking is a high-stakes arms race, and complete transparency can just make it easier for malicious users gain unfair advantages. It’s likely you’re going to want to run experiments on search ranking without disclosing them.

It would be malpractice to give users the expectation that they will always know the state of running experiments. They will not have the complete picture. Leading them to believe otherwise can do more harm to your relationship than just having a consistent policy of remaining silent until features are ready for release.

Sharing too much too soon can doom your A/B tests. But this doesn’t mean that you are doomed to be locked in a steel cage match with your user base over them.

Forum moderators of the world: good luck.

You can do rigorous, well-controlled experiments and also announce features in advance of their release. You can give people time to acclimate to them. You can let users preview new functionality, and enable them at a slower pace. These practices all relate to how a feature is released, and they are not necessarily in conflict with how you decide which features should be released. It is important to decouple these concerns.

You can and should share information about completed experiments. “What happened in the A/B test” should be a regular feature of your release notes. If you really have determined that your new functionality performs better than what it replaces, your users should have this data.

Plain-language A/B test results can ease user anxiety in launches.

Counterintuitively, perhaps, trust is also improved by sharing the details of failed experiments. If you only tell users about your victories, they have no reason to believe that you are behaving objectively. Who’s to say that you aren’t just making up your numbers? Showing your scars (as I tried to do with my homepage story above) can serve as a powerful declaration against interest.

Successful Testing is Good Stewardship

Your job in product development, very broadly, is to make progress while striking a balance between short and long term concerns.

Users should be as happy as possible in the short term.
Your site should continue to exist in the long term.

The best interest of your users is ultimately served by making the correct changes to your product. Talking about experiments can break them, leading to both quantitative errors and mistakes of judgment.

I firmly believe that A/B tests in any organization should be as free, easy, and cheap as humanly possible. After all, running A/B tests is perhaps the only way to know that you’re making the right changes. Disclosing experiments as they are running is a policy that can alleviate some discontent in the short term. But the price of this is making experiments harder to run in the long term, and ultimately making it less likely that measurement will be done at all.

Thanks to Nell Thomas, Steve Mardenfeld, and Dr. Parker for their help on this.

Growth Hacker TV Interview

2013-07-01T00:00:00+00:00

Here’s an interview I did with Growth Hacker TV last week. We covered many topics:

What exactly were you smoking when you made experimentcalculator.com?
How did Etsy get started running experiments?
Do you really hate bandit testing?
Do you really think performance never matters?
Do you really hate real time analytics?

Check it out, for the answer to these and many other thrilling questions.

Belated Network World Story

2013-06-12T00:00:00+00:00

It’s old news by now, but a few months back I was on the cover of Network World along with my colleagues Nellwyn, Steve, and Dottie.

If you missed it when it came out, you can read the article here.

My Magnum Opus (Reconsidered)

2013-06-10T00:00:00+00:00

I have been very fortunate in a number of respects. I have access to the twisted and talented mind of Eric Beug. And not only do I have a broad mandate to behave like a lunatic, but I also have dozens of like-minded coworkers. I got paid to make this. That makes me a professional actor.

You may remember my slightly-lower-budget debut.

How Long Should You Run Experiments?

2013-05-13T00:00:00+00:00

The question of how long an A/B test needs to run comes up all the time. And the answer is that it really depends. It depends on how much traffic you have, on how you divide it up, on the base rates of the metrics you’re trying to change, and on how much you manage to change them. It also depends on what you deem to be acceptable rates for Type I and Type II errors.

In the face of this complexity, community concerns (“we don’t want too many people to see this until we’re sure about it”) and scheduling concerns (“we’d like to release this week”) can dominate. But this can be setting yourself up for failure, by embarking on experiments that have little chance of detecting positive or negative changes. Sometimes adjustments can be made to avoid this. And sometimes adjustments aren’t possible.

"You ran an A/B test at one percent for a week" - the seldom-heard, missing verse of You Played Yourself.

To help with this, I built a tool that will let you play around with all of the inputs. You can find it here:

http://www.experimentcalculator.com

Here’s an example of what you might see using this tool:

The source code is available on github here. The sample size estimate in use is the one described by Casagrande, Pike and Smith.

The following people were all great resources to me in building this: Steve Mardenfeld, James Lee, Kim Bost, William Chen, Roberto Medri, and Frank Harris. Peter Seibel wrote an internal tool a while back that got me thinking about this.

The Case Against Bandit Testing

2013-01-24T00:00:00+00:00

Many have asked me if Etsy does bandit testing. The short answer is that we don’t, and as far as I know nobody is seriously considering changing that anytime soon. This has come up often enough that I should write down my reasoning.

First, let me be explicit about terminology. When we do tests at Etsy, they work like this:

We have a fixed number of treatments that might be shown.
We assign the weighting of the treatments at the outset of the test, and we don’t change them.
We pick a sample size ahead of time that makes us likely to notice differences of consequential magnitude.

In addressing “bandit testing,” I’m referring to any strategy that might involve adaptively re-weighting an ongoing test or keeping experiments running for indefinitely long periods of time.

Noel Welsh at Untyped has written a high-level overview of bandit testing, here. It’s a reasonable introduction to the concept and the problems it addresses, although I view the benefits to be more elusive than it presents. “It is well known in the academic community that A/B testing is significantly sub-optimal,” it says, and I have no reason to doubt that this is true. But as I hope to explain, the domain in which this definition of “sub-optimal” applies is narrowly constrained.

Gremlins: The Ancient Enemy

At Etsy, we practice continuous deployment. We don’t do releases in the classical sense of the word. Instead, we push code live a few dozen lines at a time. When we build a replacement for something, it lives beside its antecedent in production code until it’s finished. And when the replacement is ready, we flip a switch to make it live. Cutting and pasting an entire class, making some small modifications, and then ramping it up is not unheard of at all. Actually, it’s standard practice, the aesthetics of the situation be damned.

This methodology is occasionally attacked for its tendency to leave bits of dead code lying around. I think that this criticism is unfair. We do eventually excise dead code, thank you. And all other methods of operating a consumer website are inferior. That said, if you twist my arm and promise to quote me anonymously I will concede that yes, we do have a pretty epic pile of dead code at this point. I’m fine with this, but it’s there.

Continuous delivery: artist's conception.Photo credit: Ben Pollard

My experience here has revealed what I take to be a fundamental law of nature. Given time, the code in the “off” branch no longer works. Errors in a feature ramped up to small percentages of traffic also have a way of passing unnoticed. For practitioners of continuous deployment, production traffic is the lifeblood of working code. Its denial is quickly mortal.

This relates to the discussion at hand in that bandit testing will ramp the losers of experiments down on its own, and keep them around at low volume indefinitely. The end result is a philosophical conundrum, of sorts. Are the losers of experiments losing because they are broken, or are they broken because they are losing?

Accounting Irregularities

The beauty of Etsy’s A/B testing infrastructure lies in its simplicity.

Experiments are initiated with minimal modifications of our config file.
Visitors are bucketed based on a single, fixed-width value in a persistent cookie.

One of the advantages of this parsimony is that new tests are “free,” at least in the engineering sense of the word. They’re not free if we are measuring the mass of their combined cognitive overhead. But they are free in that there are no capacity implications of running even hundreds of experiments at once. This is an ideal setup for those of us who maintain that the measurement of our releases ought to be the norm.

Bandit testing upsets this situation in an insidious way. As I explained above, once we weight our tests we don’t tweak the proportions later. The reason for this is to maintain the consistency of what visitors are seeing.

Imagine the flow of traffic immediately before and after the initiation of an experiment on Etsy’s header. For visitors destined for the new treatment, at first the header looks as it has for several years. Then in their next request, it changes without warning. Should we attribute the behavior of that visitor to the old header or to the new one? Reconciling this is difficult, and in our case we dodge it by throwing out visits that have switched buckets. (We are not even this precise. We just throw out data for the entire day if it’s the start of the experiment.)

Bandit testing, in adjusting weights much more aggressively, exacerbates this issue. We would be forced to deal with it in one way or another.

Balancing the minimization of regret with the maximization of test adoption in your organization: artist's conception.

We could try to establish a rule for what to do with visits that see inconsistent behavior. A universally-applicable heuristic for this is not straightforward. And even if feasible, this approach would necessitate making the analysis more complicated. Increasing complexity in analysis increases the likelihood of it being incorrect.

We could continue to ignore visits that see inconsistent behavior. Depending on specifics, this could discard a large amount of data. This decreases the power of the experiment, and undermines its ability to reach a correct conclusion.

We could attempt to ensure that visits only ever see one treatment, while re-weighting the test for fresh visitors. This sounds like a great idea, but ruins the notion of tests as an operational free lunch. Test variant membership, for Etsy, is independent across web requests. Introducing dependence brings tradeoffs that developers should be familiar with. We could keep test membership in a larger cookie, but if the cookie gets too large it will increase the number of packets necessary for user requests. We could record test membership on the server, but we would have to build, maintain, and scale that infrastructure. And every time we added an experiment, we would have to ask ourselves if it was really worth the overhead.

On the Ridiculous Expectations of Runaway Victory

When we release any new feature, it is our hope that it will be a gigantic and undeniable success. Sadly (and as I have discussed at length before), this is practically never what happens. Successful launches are almost always characterized by an incremental improvement in some metric other than purchases or registrations.

Wins in terms of purchases do happen occasionally, and they make life considerably more bearable when they do. But they’re exceedingly rare. What is not rare is the experience of releasing something that makes purchase conversion worse. This turns out to be very easy in an annoyingly asymmetrical way.

What we are usually aiming for with our releases is tactical progress on our longer-term strategic goals. Modest gains or even just “not extremely broken” is what we can rationally hope for. Given this background, bandit testing would be wildly inappropriate.

Regret Approaches Zero

Let me point out something that may not be obvious: when we test features on Etsy, we are not typically testing the equivalent of banner advertisements with a limited shelf life. Not that I am suggesting that there is anything wrong with doing so. Nor do I think this is the only scenario in which bandit testing is called for.

But new features and the redesign of existing features are different in several important ways. The unlikelihood of purchase or registration conversion wins means that “regret” in the vernacular sense is minimal to begin with, obviating the need for an algorithm that minimizes regret in the technical sense. And the fact that we are building features for the longer term implies that any regret accumulated during the course of an experiment is minor from the perspective of all history. From this vantage point, the elegant simplicity of not banding testing wins out.

In Closing

Is bandit testing right for you? I believe it is a question worth asking. It may be the case that you should (to borrow Noel’s imagery) “join their merry band.” And if so, master, be one of them; it is an honourable kind of thievery.

In the absence of practical constraints, I have no argument against this. But reality is never lacking in practical constraints.

Testing to Cull the Living Flower

2013-01-16T00:00:00+00:00

I was once oblivious to A/B testing.

My first several years out of college were spent building a financial data website. The product, and the company, were run by salesmen. Subscribers paid tens of thousands of dollars per seat to use our software. That entitled them to on-site training and, in some cases, direct input on product decisions. We did giant releases that often required years to complete, and one by one we were ground to bits by long stretches of hundred-hour weeks.

Whatever I might think of this as a worthwhile human endeavor generally, as a business model it was on solid footing. And experimental rigor belonged nowhere near it. For one thing, design was completely beside the point: in most cases the users and those making the purchasing decisions weren’t the same people. Purchases were determined by a comparison of our feature set to that of a competitor’s. The price point implied that training in person would smooth over any usability issues. Eventually, I freaked out and moved to Brooklyn.

When I got to Etsy in 2007, experimentation wasn’t something that was done. Although I had some awareness that the consumer web is different animal, the degree to which this is true was lost on me at the time. So when I found the development model to be the same, I wasn’t appropriately surprised. In retrospect, I still wouldn’t rank waterfall methodology (with its inherent lack of iteration and measurement) in the top twenty strangest things happening at Etsy in the early days. So it would be really out of place to fault anyone for it.

Here is an official video produced by Etsy a few months after your author started work.

So anyway, in my first few years at Etsy the releases went as follows. We would plan something ambitious. We’d spend a lot of time (generally way too long, but that’s another story) building that thing (or some random other thing; again, another story). Eventually it’d be released. We’d talk about the release in our all-hands meeting, at which point there would be applause. We’d move on to other things. Etsy would do generally well, more than doubling in sales year over year. And then after about two years or so we would turn off that feature. And nothing bad would happen.

Some discussion about why this was possible is warranted. The short answer is that this could happen because Etsy’s growth was an externality. This is still true today, in 2013. We have somewhere north of 800,000 sellers, thousands of whom are probably attending craft fairs as we speak and promoting themselves. And also, our site. We’re lucky, but any site experiencing growth is probably in a similar situation: there’s a core feature set that is working for you. Cool. This subsidizes anything else you wish to do, and if you aren’t thinking about things very hard you will attribute the growth to whatever you did most recently. It’s easy to declare yourself to be a genius in this situation and call it a day. The status quo in our working lives is to confuse effort with progress.

An illustration of the problem of product development at Etsy, or generally when growth is an externality. Success and failure both superficially resemble success.

But I had stuck around at Etsy long enough to see behind the curtain. Eventually, the support tickets for celebrated features would reach critical mass, and someone would try to figure out if they were even worth the time. For a shockingly large percentage, the answer to this was “no.” And usually, I had something to do with those features.

I had cut my teeth at one job that I considered to be meaningless. And although I viewed Etsy’s work to be extremely meaningful, as I still do, I couldn’t suppress the idea that I wasn’t the making the most of my labor. Even if the situation allowed for it, I did not want to be deluded about the importance and the effectiveness of my life’s work.

Measurement is the way out of this. When growth is an externality, controlled experiments are the only way to distinguish a good release from a bad one. But to measure is to risk overturning the apple cart: it introduces the possibility of work being acknowledged as a regrettable waste of time. (Some personalities you may encounter will not want to test purely for this reason. But not, in my experience, the kind of personalities that wind up being engineers.)

Through my own experimentation, I have uncovered a secret that makes this confrontation palatable. Here it is: nearly everything fails. As I have measured the features I’ve built, it’s been humbling to realize how rare it is for them to succeed on the first attempt. I strongly suspect that this experience is universal, but it is not universally recognized or acknowledged.

If someone claims success without measurement from an experiment, odds are pretty good that they are mistaken. Experimentation is the only way to seperate reality from the noise, and to learn. And the only way to make progress is to incorporate the presumption of failure into the process.

Don’t spend six months building something if you can divide it into smaller, measurable pieces. The six month version will probably fail. Because everything fails. When it does, you will have six months of changes to untangle if you want to determine which parts work and which parts don’t. Small steps that are validated not to fail and that build on one another are the best way, short of luck, to actually accomplish our highest ambitions.

To paraphrase Marx: the demand to give up illusions is the demand to give up the conditions that require illusions. I don’t ask people to test because I want them to see how badly they are failing. I ask them to test so that they can stop failing.

Yes! The Deploy Dashboard Graphs “Screwed Users.”

2013-01-11T00:00:00+00:00

In my post about real-time analysis I shared a screenshot of part of Etsy’s deployment dashboard. This is the dashboard that every engineer watches as he or she pushes code to production. A bunch of alert readers noticed some odd things about it:

The screenshot is not doctored, so yes we do graph “Three-Armed Sweaters” and “Screwed Users.” I can explain. In fact, I can give you excruciating detail about it, if you’re interested! Here goes.

“Three-Armed Sweaters” refers to our error pages, which feature one of my favorite drawings in the world. It was done by Anda Corrie:

Although purely theoretical at first, real versions of the sweater have since been commissioned. These are handed out yearly to the Etsy engineer that [brings the site down in the most spectacular fashion](http://bits.blogs.nytimes.com/2012/07/18/one-on-one-chad-dickerson-ceo-of-etsy/).

So the graph on the dashboard is just counting the number of times this page is shown. But in order to reduce the frequency of false alarms, the graph is actually based on requests to an image beacon hidden on the page. This excludes most crawlers and vulnerability scanners. Those constituencies have a habit of generating thousands of errors when nothing is malfunctioning. But lucky for us, they almost never waste bandwidth on images.

Now, there are many reasons why Etsy might not be working, and they don’t all result in our machines serving a sweater page. If our CDN provider can’t reach our production network, it will show an error page of its own instead. In these cases, our infrastructure may not even be seeing the requests. But we can still graph these errors by situating their image beacon on a wholly separate set of web machines.

The “screwed users” graph is the union of all of these conditions. So-called, presumably, because all of this nuance is relatively meaningless to outsiders. “Screwed users” also attempts to only count unique visitors over a trailing interval. This has the nice property of causing the screwed users and sweaters graphs to diverge in the event that a single person is generating a lot of errors. The internet, after all, is full of weird people who occasionally do weird things with scripts and browsers.

This is what it looks like when many real users are seeing error pages. Both graphs spike in concert. In this case, the dark green vertical line shows a search deploy resolving the issue.

You now know exactly as much as I do about the graphing of web errors in real time. I assume that this is a tiny fraction of the world’s total knowledge pertaining to the graphing of web errors in real time. So you would be ill-advised to claim expert status on the basis of grasping everything I have explained here.

By the way, most of the software Etsy uses to produce these graphs is freely available. Here’s StatsD and Logster.

Whom the Gods Would Destroy, They First Give Real-time Analytics

2013-01-09T00:00:00+00:00

Homer: There's three ways to do things. The right way, the wrong way, and the Max Power way!

Bart: Isn't that the wrong way?

Homer: Yeah. But faster!

- "Homer to the Max"

Every few months, I try to talk someone down from building a real-time product analytics system. When I’m lucky, I can get to them early.

The turnaround time for most of the web analysis done at Etsy is at least 24 hours. This a ranking source of grousing. Decreasing this interval is periodically raised as a priority, either by engineers itching for a challenge or by others hoping to make decisions more rapidly. There are companies out there selling instant usage numbers, so why can’t we have them?

Here’s an excerpt from a manifesto demanding the construction of such a system. This was written several years ago by an otherwise brilliant individual, whom I respect. I have made a few omissions for brevity.

We believe in…

Timeliness. I want the data to be at most 5 minutes old. So this is a near-real-time system.

Comprehensiveness. No sampling. Complete data sets.

Accuracy (how precise the data is). Everything should be accurate.

Accessibility. Getting to meaningful data in Google Analytics is awful. To start with it’s all 12 - 24 hours old, and this is a huge impediment to insight & action.

Performance. Most reports / dashboards should render in under 5 seconds.

Durability. Keep all stats for all time. I know this can get rather tough, but it’s just text.

The 23-year-old programmer inside of me is salivating at the idea of building this. The burned out 27-year-old programmer inside of me is busy writing an email about how all of these demands, taken together, probably violate the CAP theorem somehow and also, hey, did you know that accuracy and precision are different?

But the 33-year-old programmer (who has long since beaten those demons into a bloody submission) sees the difficulty as irrelevant at best. Real-time analytics are undesirable. While there are many things wrong with our infrastructure, I would argue that the waiting is not one of those things.

Engineers might find this assertion more puzzling than most. I am sympathetic to this mindset, and I can understand why engineers are predisposed to see instantaneous A/B statistics as self-evidently positive. We monitor everything about our site in real time. Real-time metrics and graphing are the key to deploying 40 times daily with relative impunity. Measure anything, measure everything!

Part of the deploy dashboard at Etsy. We love up-to-the-minute graphs.

This line of thinking is a trap. It’s important to divorce the concepts of operational metrics and product analytics. Confusing how we do things with how we decide which things to do is a fatal mistake.

So what is it that makes product analysis different? Well, there are many ways to screw yourself with real-time analytics. I will endeavor to list a few.

The first and most fundamental way is to disregard statistical significance testing entirely. This is a rookie mistake, but it’s one that’s made all of the time. Let’s say you’re testing a text change for a link on your website. Being an impatient person, you decide to do this over the course of an hour. You observe that 20 people in bucket A clicked, but 30 in bucket B clicked. Satisfied, and eager to move on, you choose bucket B. There are probably thousands of people doing this right now, and they’re getting away with it.

This is a mistake because there’s no measurement of how likely it is that the observation (20 clicks vs. 30 clicks) was due to chance. Suppose that we weren’t measuring text on hyperlinks, but instead we were measuring two quarters to see if there was any difference between the two when flipped. As we flip, we could see a large gap between the number of heads received with either quarter. But since we’re talking about quarters, it’s more natural to suspect that that difference might be due to chance. Significance testing lets us ascertain how likely it is that this is the case.

A subtler error is to do significance testing, but to halt the experiment as soon as significance is measured. This is always a bad idea, and the problem is exacerbated by trying to make decisions far too quickly. Funny business with timeframes can coerce most A/B tests into statistical significance.

A simulation of flipping two fair coins. In the green regions, the difference in the number of heads is measured to be significant. If we stopped flipping in those regions, we would (incorrectly) decide the coins were different.

Depending on the change that’s being made, making any decision based on a single day of data could be ill-conceived. Even if you think you have plenty of data, it’s not farfetched to imagine that user behavior has its own rhythms. A conspicuous (if basic) example of this is that Etsy sees 30% more orders on Tuesdays than it does on Sundays.

Gratuitous infographic courtesy Brendan Sudol.

While the sale count itself might not skew a random test, user demographics could be different day over day. Or very likely, you could see a major difference in user behavior immediately upon releasing a change, only to watch it evaporate as users learn to use new functionality. Given all of these concerns, the conservative and reasonable stance is to only consider tests that last a few days or more.

One could certainly have a real-time analytics system without making any of these mistakes. (To be clear, I find this unlikely. Idle hands stoked by a stream of numbers are the devil’s playthings.) But unless the intention is to make decisions with this data, one might wonder what the purpose of such a system could possibly be. Wasting the effort to erect complexity for which there is no use case is perhaps the worst of all of these possible pitfalls.

For all of these reasons I’ve come to view delayed analytics as positive. The turnaround time also imposes a welcome pressure on experimental design. People are more likely to think carefully about how their controls work and how they set up their measurements when there’s no promise of immediate feedback.

Real-time web analytics is a seductive concept. It appeals to our desire for instant gratification. But the truth is that there are very few product decisions that can be made in real time, if there are any at all. Analysis is difficult enough already, without attempting to do it at speed.

Dan McKinley

Some Recent Work

The Unreasonable Effectiveness of Mathematics in Planning

Data Exists, and We Don’t Want to Look

The Hazards of Narrative Arc

Do You Work at Amazon?

The Odds of Being an Amazon

Recovery is not Good Enough

Are You Making Progress?

The Base Rate Fallacy's Perverse Tyranny Over the American [4] Mind

Are My Push Notifications Driving Users Away?

How would we do this?

An example result

I’ve run the experiment, now what?

I don’t think I have enough data.

I’m sending reactivation notifications to inactive users. Can I still measure uninstalls?

I want to learn more about this stuff.

Choose Boring Technology (Expanded, Slide-Based Edition)

Choose Boring Technology

Embrace Boredom.

Optimize Globally.

Choose New Technology, Sometimes.

Just Ship.

Data Driven Products: Lean Startup 2014

Thoughts on the Technical Track

To Review, Management is a Job

Management Just Happens

Technical Track Promotions are Post-Hoc

Stumping

Society Applies to Idealistic Tech Companies, Too

Halfhearted Managers are The Worst

Ratios Observed in the Wild Make No Sense

Promotions Should Not Be Miraculous and Rare

Address Prestige with Superpowers

Data Driven Products Now!

Manual Delivery

Scalding at Etsy

The Case for Secrecy in Web Experiments

The Diamond-Forging Pressure to Disclose Experiments

Online Protests: Not Necessarily A Waste of Time

Predicting Bias is Hard

Communication Plans are Real Work

Your Daily Reminder that Experiments Fail

Consistency and Expectations

What can you share?

Successful Testing is Good Stewardship

Growth Hacker TV Interview

Belated Network World Story

My Magnum Opus (Reconsidered)

How Long Should You Run Experiments?

http://www.experimentcalculator.com

The Case Against Bandit Testing

Gremlins: The Ancient Enemy

Accounting Irregularities

On the Ridiculous Expectations of Runaway Victory

Regret Approaches Zero

In Closing

Testing to Cull the Living Flower

Yes! The Deploy Dashboard Graphs “Screwed Users.”

Whom the Gods Would Destroy, They First Give Real-time Analytics