Dan McKinley
Math, Programming, and Minority Reports

Thoughts on the Technical Track
December 9th, 2014

I saw lizTheDeveloper’s post about technical leadership at Simple and I realized that I’ve been meaning to write about this for a while. I hope to persuade you that there are a number of systemic biases working against a healthy technical career path. I don’t think that they’re insurmountable, and I don’t disagree with Liz’s post. But I’ve never heard of a company clearing all of these hurdles at once.

I was the first person at Etsy with the title of “Principal Engineer,” which was the technical equivalent to a directorship (i.e., one level below CTO). I’m not saying this to toot my own horn, but rather so that it’s understood that the following comes from someone that was the beneficiary of an existing system.

(Incidentally, I think Etsy is an example of a company whose heart is in the right place, and it’s not my intention to single them out.)

To Review, Management is a Job

My views on the merits of having a technical track align with those of many people in our industry. Management is a different job, with different skills. They’re not necessarily more difficult skills, they’re just different. By and large they’re unrelated to the day-to-day labor of the people who build technology products.

It doesn’t make any sense to divert your technical talent into a discipline where they will need to stop doing technical work. (That’s in the event that they intend to be effective managers, which I concede might be an unrealistic expectation.)

Other people have made this case, so I’ll just proceed as if we agree that there must be a way forward for people that are great programmers other than to simply graduate into not programming at all.

Having that way forward is an ideal. There is always a gap between our ideals and reality, and we cannot act as though we’ve solved a problem simply by articulating it.

Fundamental Asymmetries

Management Just Happens

I have had management responsibility thrust upon me at least four times over the course of my career, and at no point has that been my goal. It just happens. Do you want to be a manager? I will now tell you the secret to becoming a manager in a growing company: just wait.

You have a manager. Eventually, your manager will accrue too many responsibilities, and they will freak out. They will need somebody to take over some of their reports, and that lucky warm body is you.

It is entirely plausible to become a manager accidentally. It might even be the norm.

Technical Track Promotions are Post-Hoc

The process for minting a new manager is: crap, we need another manager. There’s no symmetrical forcing function pushing people into the upper ranks of technical leadership.

Mentorship and technical feedback are things everyone does on a functioning engineering team. A technical track “promotion” is merely additional recognition given to someone who is already performing that role notably well.

If the job is already getting done, then filling the job is clearly not a pressing need. Technical promotions are something that happen when it’s convenient, which is generally never.

Stumping

Between the founding of the United States and the end of the 19th century, it was considered tacky for presidential candidates to personally campaign for the job. Instead, they staged an elaborate farce in which they reluctantly answered the call of the nation to serve. Trying to intentionally get a promotion into the technical track is pretty much just like this.

Your work must be recognized, and this is the rub. Let me rephrase: “someone with the power to bestow promotions has to be your fan.” To be promoted you have to be a good mentor, but you also have to worry about playing to an audience. That may be executives, or it may be your peers (and potential competitors). Regardless, you’re running a weird campaign in which actually saying anything directly about wanting the job would be gauche.

The most qualified individual contributors may become known without ever really doing this on purpose, but that doesn’t say much for this as a tenable career goal of the sort that can be counted on.

The Problem of Credibility

Society Applies to Idealistic Tech Companies, Too

American society is not a classless oasis. That’s a lie we tell ourselves. And the person who knows what everyone else gets paid and can fire you is not in your class.

A technical job does not have equivalent prestige to a management position with an equivalent salary just because you say it does. Even if you conquer this within your own company, it’s not true in the rest of the industry, and it’s not true in the world at large. In the world our parents live in, it’s a big deal to be somebody else’s boss.

You’re hiring people from the world at large all the time. Without continuous effort a technical track decays to its ground state, where the jobs are second class.

Halfhearted Managers are The Worst

The natural result of a system in which technical promotions can’t be counted on and are viewed as suspiciously-maybe-second-class anyway is that people who don’t really give a shit about management wind up going into management. Given the choice of waiting for a technical promotion that may never arrive and taking an offer to manage others, almost everyone is going to take the bird in the hand.

Managers that have no passion for management are a blight on society. I can say this because I have been one of them. I was never a good manager, and for that I apologize to anyone that ever had to report to me.

I am not an isolated case. Many people in management are frankly terrible at it. And they would rather have technical track jobs anyway, but they have no idea how to make the switch. A credible technical track is a great way to ensure a higher level of satisfaction and competency among the managers.

Ratios Observed in the Wild Make No Sense

You don’t need to take my reasoning about the intrinsic pressure favoring management bloat at face value. You can actually look at the ratio of managers to technical employees at your company.

At one point, I was alone at my level. There were five theoretically-equivalent directors at the time. The ratio was at least that bad on the lower rungs. (I have no idea if this is still true at that company, and it might not be.)

For that to make sense, we’d have to believe a few things that don’t stand up to scrutiny. First, we’d have to believe in a very high proclivity among engineers to manage, and I think that betrays our expectations. Not very many of us got into this business with the hope of not actually building things.

Second, we’d have to believe that although it took five directors to effectively manage the organization, only one technical leader was required to advise the same group on the details of the work they do every day.

What Might Help?

Promotions Should Not Be Miraculous and Rare

Of course, it wouldn’t make logical sense to say that the ratio of individual contributors to managers at a given level must be 1:1. I honestly don’t know if 1:2 or 2:1 is closer to correct. The answer is probably contingent, and the relationship might not be linear.

But I think it’s important for any company that takes the ideal of having a tenable technical track seriously to put a stake in the ground on this question. It’s hard to build a credible technical track, and we need a baseline to grade ourselves against.

I don’t think that proceeding with the assumption that leaders will just naturally emerge produces the best results. Adding a self-imposed quota achieves accountability. It acknowledges the possibility that problems can lie in the system of recognition, and not only in the talents of the people in the pool for promotions.

“Do we think that we hire smart people here? Yes? Then we should be able to find N of them worthy of promotion for every manager. If we can’t then the problem is most likely to be found in how we’re recognizing people for their work.”

I know that the word “quota” is verboten for many, and I gleefully await your flames.

Address Prestige with Superpowers

If we think about why managers and technical employees on even salary footing may be perceived to not truly be equals, it comes down to superpowers. The managers have special capabilities that the technical employees don’t: hiring, firing, compensation, and the like. Is it possible to give technical employees a different set of superpowers, to address the prestige problem?

Maybe. I don’t think that I have seen this done correctly yet. If I had superpowers, they were:

• The ability to work on whatever I wanted.
• The ability to talk to anyone I wanted.

These were indeed powerful, but using them to create positive action was difficult. It would have been easy for me to opt out of projects that I didn’t believe in and to do my own thing. I did often do my own thing. But I also worked on projects that I didn’t believe in, because I knew that opting out was a selfish act. One of my friends would just be forced to work on it in my place, and sometimes leadership is about jumping on grenades.

Talking to other teams made it possible for me to point out places where resources weren’t intelligently allocated. But this also begat mostly negative actions. “Hey, this isn’t the best way to use these folks,” I’d find myself saying all the time. It was draining, and a bummer.

Giving the technical leadership deeper involvement in the planning process could address this. Of course that would involve dragging the technical leadership to meetings, which I admit is tricky.

In Closing

I hope I’ve demonstrated that creating a career path outside of management for technical employees is only the beginning of your problems. It’s a good and necessary step, but it’s not an achievement by itself.

I’d love to hear from anyone with better ideas. These issues are difficult and I don’t claim to have all of the right answers.

Data Driven Products Now!
September 18th, 2014

Back when I was at Etsy, I did a presentation internally about the craft of sizing opportunities. I finally got around to writing a public incarnation of that talk. Here it is:

Manual Delivery
March 10th, 2014

The person on build rotation, or the nightly schlimazel I suppose, went into a hot 5’x8’ closet containing an ancient computer. This happened after everyone else had left, so around 8:30PM. Although in crunch time that was more like 11:30PM. And we were in crunch time at one point for a stretch of a year and a half. “That release left a mark,” my friend Matt used to say. In a halfhearted attempt at fairness to those who will take this post as a grave insult, I’ll concede that my remembrance of these details is the work of The Mark.

Anyway, the build happened after quitting time. This guaranteed that if anything went wrong, you were on your own. Failure in giving birth to the test build implied that the 20 people in Gurgaon comprising the QA department would show up for work in a matter of hours having nothing to do.

You used a tool called “VBBuild.” This was a GUI tool, rumored to be written by Russians:

VBBuild did mysterious COM stuff to create the DLLs that nobody at the time understood properly. It presented you with dozens of popups even when it was working perfectly, and you had to be present to dismiss each of them. The production of executable binary code was all smoke and lasers. And, apparently, popups.

Developers wrote code using the more familiar VB6 IDE. The IDE could run interpreted code as an interactive debugger, but it could not produce finished libraries in a particularly repeatable or practical way. So the release compilation was different in many respects from what programmers were doing at their desks. Were there problems that existed in one of these environments but not the other? Yes, sometimes. I recall that we had a single function that weighed in at around 70,000 lines. The IDE would give up and execute this function even if it contained clear syntax errors. That was the kind of discovery which, while exciting, was wasted in solitude somewhere past midnight as you attempted to lex and parse the code for keeps.

Developers weren’t really in the habit of doing complete pulls from source control. And who could blame them, since doing this whitescreened your machine for half an hour. They were also never in any particular hurry to commit, at least until it was time to do the test build. As there was no continuous integration at the time, this was the first time that all of the code was compiled in several days.

Often [ed: always] there were compilation errors to be resolved. We were using Visual Sourcesafe, so people could be holding an exclusive lock on files containing the errors. Typically, this problem was addressed by walking around the office an hour before build time and reminding everyone to check their files in. In the event that someone forgot [ed: every time], there was an administrative process for unlocking locked files. Not everyone had the necessary rights to do this, but happily, I did.

By design, the build tried to assume an exclusive lock on all of the code. As a result, nobody could work while the build was in progress. Sometimes, the person performing the build would check all of the files out and not check them back in. So your first act the morning after a build might be to walk over to the build closet and release the source files from their chains.

Deployment required dozens of manual steps that I will never be able to remember. When the build was done, you copied DLLs over to the test machines and registered them there. By “copied” I mean that you selected them in an explorer window, pressed “Ctrl-C,” and then pressed “Ctrl-V” to paste them into another. There was no batch script worked out to do this more efficiently. Ok, this is a slight lie. There had been a script, but was put out to pasture on account of a history of hideous malfunction. And popups. On remote machines sometimes, where they could only be dismissed by wind and ghosts.

Registration involved connecting to each machine with Remote Desktop and right clicking all the DLLs. You could skip a machine or just one library, and things would be very screwy indeed.

The production release, which happened roughly twice a year under ideal conditions, was identical to this but with the added complexity of about eight more servers receiving the build. And we might take the opportunity to add completely new machines, which would not necessarily have the same patch levels for, oh, like 700,000 windows components that were relied upon.

Given eight or ten machines, the probability of a mistake on at least one of the servers approached unity. So the days and weeks following a production release were generally spent sussing out all of the minute differences and misconfigurations on the production machines. There would be catastrophic bugs that affected a tiny sliver of requests, under highly specific server conditions, and only if executed on one server out of eight. I was an expert at debugging in disassembly at the time. Upon leaving the job, I thought that this was pretty badass. But in the seven years since–do you know what? It’s never come up.

At one point I wrote a new script to perform the deployment. It was an abomination of XML to be sure, but it got the job done without all of the popups. I started doing the test build with this with some success and suggested that we use it for the production release. This was out of the question, I was told by one of my closer allies in the place. The production release was “too important to use a script.”

The operating systems and supporting libraries on the machines were also set up by hand, by a separate team, working from printed notes. The results were similar. This is kind of another story.

This all happened in 2003.