The Principal Engineer Paradigm

Introduction

As a Sr Engineer in Big Tech, I’ve been working with Principal Engineer the past month or so. It struck me the extento which the role is distinct role from the Sr, not that they are execute a Sr’s functions all the better. In this post I’m using ‘Principal’ based on the role’s level at Amazon. Other companies may call it Staff, or Sr Staff, but you will recognize it when you see it. It’s the top dawg on the technical reporting chain.

What Needs To Be Done?

Sr. Engineers are trusted to lead designs. They pull off the how. Principal Engineers don’t add a lot of value by designing things better - the Sr Engineer’s designs should already be up to snuff. But Principal Engineers dig deep into why the hell are we doing this particular task!? Does it really need to be done to accomplish the goal ? Can this step be simplified? Can it be removed? Principals keep Srs from reinventing things and from choosing suboptimal tools from an org’s perspective. Even if its beened signed off on by various stakeholders, engineering managers, and cross-team collaborators, a Principal Eng can come in and question your project’s objective, your tools, timelines, and often your sanity.

Are You Sure You Aren’t Wrong?

Your manager may question your methods, but they’ll give you the benefit of the doubt. Over the weeks and months they’ve seen your work and heard your status updates. They trust that you’ll ultimately deliver and don’t spend too much time nit-picking your decisions. A Principal Engineer gives you no such quarter. They ruthlessly question your timeline estimates, your project plans, whether you have thought enough about security, scaling,reliability, and whether you have sufficient test coverage. To err is human, and a PE’s job is save you from this natural inclination.

Isn’t This A PM’s Job?

A Principal Engineer and a Product Manager both have a main goal of making a project succeed. But the PE brings a technical perspective. They can challenge your estimates from a position of equal or superior technical knowledge as well as a superior understanding of your organization’s overall strategic approach to software. A PM might question your timelines, but only a PE can refute them.

Owning The Standards

If your software team builds one giant application, the Principal may make the highest-level decisions on your architecture:
monolith vs microservices, multi-tenant vs single, bring your own container vs managed Kubernetes, etc. If the team builds many indpendent services, he may decide its standard tooling. Either way, it’s a way different responsibility level than leading a project. Decisions must be optimal given multiple systems and sets of engineers. What’s optimal for the organization level may be may be very suboptimal for an individual project or team, and the engineers involved may strongly resent the decision. Crucially, if things are going truly wrong, there is also no room to back up and change one’s mind. A project that is on a bad path can get back on track by engineers resolving to spend a few weekends grinding to get the project back on track. The PE-level decisions offer no such possibilities. PE decisions decide sustantial time and money investments, which once made can be the difference between the end product ultimately succeeding or failing. At decision time, it’s lonely at the top.

Is Isn’t Personal

Good principal engineers have backbone. They may defend a data governance policy against two separate teams who have already agreed upon granting one-time exception. They may challenge and force a reversal on a well-debated technical decison. Invariablly, if you are an engineer making consequential decisions, the PE will outright annoy you at some point. But if they never make you unhappy or uncomfortable, they aren’t an effective Principal.

Building This Skill

A lot of us got into tech for the love of puzzles, the thrill of finding things out, and the commradarie of building things. A good PE experiences little of that. He spends most of his time above the weeds, doesn’t stay on a single project, and frequently pisses off engineering managers and Sr Engineers. Yet, if you want to make the biggest technical impact you can, you must develop this separate set of skills. The Staff Engineer's Guide by Tanya Reilly is a good resource to explore the role in depth. But to fully understand their purpose it helps to experience a challenge from a Principal Engineer in your own work. Because as Seraph says in The Matrix Reloaded:

You Never Really Know A Man Until You Fight Him - Seraph