What I’ve been reading

  • Christopher Hitchens, Why Orwell Matters. A look at Orwell in the context of his time, and how and why he holds up so well. What struck me is what Hitchens calls Orwell’s “power of facing,” in other words, the ability to look directly at unpleasant facts.
  • Charlie Jane Anders, The City in the Middle of the Night. I didn’t much care for her previous work, All the Birds in the Sky. This one is a great read. It also has a great title.
  • Walter Miller, A Canticle for Leibowitz. A classic.
  • Daniel Todman, Britain’s War: A New World, 1942-1947. Covers a lot of ground that previous books I’ve read about WWII haven’t. War is very costly.

Elsewhere on the web

Pinboard is Eleven. “Much of the core code on the site dated back to 2009-2010 and was written by Past Me, a vindictive, inscrutable nemesis who devoted his life to sabotaging Present Me. Doing this on a live system is like performing kidney transplants on a playing mariachi band. The best case is that no one notices a change in the music; you chloroform the players one at a time and try to keep a steady hand while the band plays on. The worst case scenario is that the music stops and there is no way to unfix what you broke, just an angry mob. It is very scary.”

How to Write Technical Posts (so people will read them). “Here’s the biggest thing to keep in mind: your reader doesn’t really care what you have to say.”

Recognizing vs Generating. “Here, as in many other Recognizing vs Generating dichotomies, it can be easy to trick ourselves into thinking we’re putting in more effort than we really are. “After all,” the tempting thought goes, “both reading the textbook and writing proofs feel like they fit the definition ‘studying’, so why can’t the easier one work?”

Narrative Collapse. “In other words, when you read a narrative, for example, you are encountering the product of a series of choices that have already been made for you by the author out of a myriad of possibilities… The countless other choices that were possible are present only to the imagination. You see the words the author chose, not the ones she could’ve chosen. You see the path marked out for you as a reader, not the multiple paths that were rejected.”

Notes on The Art of Leadership: Small Things, Done Well by Michael Lopp

The Art of Leadership Cover

I owe a lot to Michael Lopp. Years ago, his book, Being Geek, and his blog, Rands in Repose, were my first introductions to thinking about software engineering and engineering management as a capital letter “C” Career. And they were my first introduction to the concept of regular 1:1s and other fundamental management concepts. What I learned from him allowed me to avoid creating a total disaster of everything when I was made a manager with no training. Where I am in my career right now, I can trace back to what I first started learning from Lopp, and I am still learning from him today.

(Thank you!)

That is all to say that I recommend The Art of Leadership: Small Things, Done Well (and his other writing) to anyone in software engineering management, or considering software engineering management. Most of the book is adapted directly from Rands in Repose, so you could get some value from browsing the archive. That said, he’s selected the best material, refined it, and added entirely new chapters, so I recommend reading the book.

Lopp’s signature technique is to tell a story to illustrate his point. This technique is compelling, engaging, and works well.

Most useful chapters

I found the following chapters most useful:

  • 2. Meeting Blur
  • 7. A Performance Question
  • 8. How to recruit
  • 9. Rainbows and Unicorns
  • 10. Say the Hard Thing
  • 11. The Signal Network
  • 12. Be Unfailingly Kind

Resonating concepts

The following concepts resonated the most with me:

  1. Information dissemination is a core responsibility of a manager.
  2. A manager should set the highest standard for follow-through.
  3. Recruiting is a core responsibility of a manager.
  4. Compliments work.
  5. Say the hard thing. Your goal should be to make feedback routine.
  6. As a leader, it’s ok to selectively lean on your experience instead of working to gain consensus.

These concepts are critical. So much so that I hope to write about many of these separately, bringing in other sources. For now, I’ll briefly discuss what Lopp says about each, and use each as a vehicle for me to expand a bit.

1. Information dissemination is a core responsibility of a manager

My educated guess is that 50% of my job as a manager is information acquisition, assessment, and redistribution. It is my primary job, and the efficiency with which I do this directly contributes to the velocity of the team.

You can spend a lot of time and money investing in processes, tools, and artifacts that you believe are necessary to critical and timely information flow, but where I consistently invest is in the team. I demonstrate to the humans the value of effectively detecting, assessing, routing, and retransmitting information across the organization.

A manager has access to a lot of information that the team doesn’t have about what’s going on elsewhere in the company, for example in a high-profile meeting. As a leader, you have a responsibility to communicate this to the team to help them work more effectively.

I’ve been thinking a lot about this one lately. Communication is the challenge of a growing team. The number of possible 1-1 communication channels increases proportionally to the square of people. As your organization grows, you start to need new layers in the form of middle managers to tie things together, which introduce even more communication channels—and opportunities for communication to not happen.

Ideally, you can organize teams so that the right people naturally talk to each other, establish mature project management, organize structurally recurring meetings (daily standups, etc.), and foster a culture of communicating well in writing through tickets. But… as experience shows, there are always gaps. It’s a manager’s job to actively seek out and fill those gaps.

2. A manager should set the highest standard for follow-through

“You sign up for things and get them done. Every single time.”

I liked this so much that I already quoted it. A big part of being a manager is nudging people to the behavior you want through setting an example of that behavior. Follow-through, in particular, is so fundamental because it is the foundation of trust. You do what you say. What you say means something. People can rely on you.

3. Recruiting is a core responsibility of a manager

On the list of work you can do to build and maintain a healthy and productive engineering team, the work involved in discovering, recruiting, selling, and hiring the humans for your team is quite likely the most important you can do.

Lopp advocates spending at least one hour per day on recruiting-related activities, up to 50% of your time. That’s a lot! But, this is something I believe in and have advocated for myself. Will Larson has written about spending time each week on cold sourcing.

The critical point here is that growing the team, in the right way, is the manager’s responsibility (and I would argue accountability). The composition of the team is one of the most significant factors that will determine the team’s future performance. Too many managers rely on HR to do everything in recruiting, but that’s an abdication of responsibility.

You can rely on you HR/recruiting team as a powerful partner to do the actual work of traditional screening, recruiting, etc. but you need to be involved. Since you are closer to the work, you are better positioned to have the meaningful conversations with candidates that can truly identify who the best match is, and more importantly, communicate to the candidate in their language why they should want to work at your company, and with you, personally.

Lopp identifies three stages of a recruiting pipeline for engineering managers, which differs from the traditional recruiting pipeline:

  1. Discover. Networking and cold sourcing.
  2. Understand. During the candidate interview process, your goal here is to make sure that the candidate understands your mission, culture, and values.
  3. Delight. From offer to onboarding. Just because a candidate accepted an offer doesn’t mean that they will show up on the first day. After a candidate accepts an offer is a dangerous time, where the reality of impending change hits them, and doubt and uncertainty are at their most intense. Stay engaged with the new hire through meaningful, personal communication. Describe to them what they will be working on. Don’t rely on HR to be the sole communicators with the candidate.

4. Compliments work

People like compliments; they make them feel good. It is a form of timely feedback that helps people learn. Lopp gives a useful, practical definition of a compliment as a “a selfless, well-articulated, and timely recognition of achievement.”

All of those components are important, but the most important to me is “well-articulated.” Don’t just say “good job.” That’s easy to toss off, and the recipient knows it. It’s cheap. Specifically state the act, the value, and the impact. It shows your paying attention, and it communicates what you value, and what the team values.

This resonates with me because frankly, like most managers, I don’t do this enough.

5. Say the hard thing. Your goal should be to make feedback routine.

Feedback is an incredibly valuable social transaction. It shows that people have taken their time to observe an aspect of you. They have other things to do, but today they are investing in you. You think you’ve got it all figured out, but you don’t. In turn, you take the time to clearly hear the feedback, ask clarifying questions, and hopefully adjust the way you work. All the constituent parts of the act of giving and receiving feedback provide an opportunity to build trust in a relationship.

There are two parts to this. First, say the hard thing. Since reading this book, I’ve kept this phrase in my head.

The ability to say the hard thing is a critical leadership skill. It is essential to helping your team grow. Positive feedback (compliments!) can go a long way, but sometimes you have to tell someone something they don’t want to hear. It’s easy not to say the hard thing because we don’t want to hurt someone’s feelings. We know that we wouldn’t want to hear it, so we don’t want to say it. However, as I have learned from personal experience, the only thing worse than saying the hard thing is not saying it.

Another way I like to think about this is that you should tell people what they need to hear, not what they want to hear. More bluntly: avoiding a hard conversation is selfish, because you are putting your own emotional comfort above your responsibility to help the person grow, your responsibility to other team members who deserve the best teammates, and your responsibility to your employer to guide the team to achieve great performance.

The second part is making feedback—saying the hard thing—routine, so it is a bit less hard to say. It’s an essential behavior to model and foster. It takes practice, and it isn’t easy. A team that can talk to each other candidly about what is going well, and what is not, is a team that trusts each other, and a team that will do greater and greater things.

6. As a leader, it’s ok to selectively lean on your experience instead of working to gain consensus

This is a bit of a corollary to avoiding the  “not invented here” syndrome.

As you advance in your career, one advantage you gain from experience, and from being an executive, is that you can selectively draw on that experience to shortcut what would potentially be a long discovery and decision-making process. The example he uses is that he adapted a career latter from a previous gig instead of working collaboratively with the team to create a new one. It wasn’t necessarily as good of a fit as a custom one would have been, but it was much faster to produce. The team was able to focus on higher-value work.

Conclusion

The Art of Leadership is valuable for both new and experienced engineering managers for different reasons. For the new manager, these concepts and the stories that frame them can help form some of the mental matrix you use to orient yourself as you face new situations. For the experienced manager, Lopp has a way with words and a way with framing that can help clarify your thinking, even if you already know the specific concept under discussion.

MailMate and Alfred

I use MailMate for my email, since it is very fast, customizable, and integrates with many other programs, and supports Markdown natively. It isn’t pretty, its documentation has gaps, and it isn’t for everyone—but it’s power cannot be matched. I’m a big believer in minimizing the amount of time it takes to do common tasks. I can work faster and more easily keep my mental context. Here are a few Alfred integrations I’ve created to make my work faster and keep my hands on the keyboard.

I created an Alfred keyword (“m”) to search MailMate messages using its search syntax, using it’s mlmt: URI scheme.

MailMate Alfred workflow screenshot

The bash script is simple:

query=$1
open "mlmt:quicksearch?string=$query"

So I can do the following:

Example of mailmate search in Alfred

In other words, look for messages from Sam that have “project rhino” in the subject.

I like searching this way because the syntax is powerful, but I can immediately search for what I need, from anywhere, as soon as I need to, in a single action.

Open inbox and go to first unread message

Alfred’s AppleScript Dictionary is rudimentary but contains one powerful command: perform. This allows you to perform any MailMate action that you can do with custom key bindings, which is a lot.

I created an inbox keyword that opens up the inbox, and immediately scrolls to the first unread message and opens it. When I see a notification for an incoming message that I care about, I can simply type “inbox” into Alfred and jump immediately to it.

Example of MailMate inbox workflows

on alfred_script(q)
	tell application "MailMate"
		perform {"goToMailbox:", "INBOX"}
		perform {"nextUnreadMessage:"}
	end tell
end alfred_script

I also have a pinbox keyword that jumps to a “personal” smart mailbox that has only inbox messages addressed directly to me from people who I have emailed in the past (the UUID uniquely identifies the smart mailbox):

on alfred_script(q)
	tell application "MailMate"
		perform {"goToMailbox:", "5E738709-814A-4DFA-93E5-85BB6FECD3F8"}
		perform {"nextUnreadMessage:"}
	end tell
end alfred_script

They will be our witnesses when we are gone

Seven years back, when Florence was besieged by the Emperor and begging for French aid, the burgesses went to the merchant Borgherini’s house: ‘We want to buy your bedroom.’ There were fine painted panels, rich hangings and other furnishings they thought might make a bribe for King François. But Margherita, the merchant’s wife, stood her ground and threw their offer back in their faces. Not everything in life is for sale, she said. This room is my family’s heart. Away with you! If you want to take away my bedroom, you will have to carry your loot over my corpse.

He would not die for his furniture. But he understands Margherita—always supposing the story to be true. Our possessions outlast us, surviving shocks that we cannot; we have to live up to them, as they will be our witnesses when we are gone. In this room are the goods of people who can no longer use them. There are books his master Wolsey gave him. On the bed, the quilt of yellow turkey satin under which he slept with Elizabeth, his wife. In a chest, her carved image of the Virgin is cradled in a quilted cap. Her jet rosary beads are curled inside her old velvet purse. There is a cushion cover on which she was working a design, a deer running through foliage. Whether death interrupted her or just dislike of the work, she had left her needle in the cloth. Later some other hand—her mother’s, or one of her daughters’—drew out the needle; but around the twin holes it left, the cloth had stiffened into brittle peaks, so if you pass your finger over the path of her stitches—the path they would have taken—you can feel the bumps, like snags in the weave. He has had the small Flanders chest moved in here from next door, and her furred russet gown is laid up in spices, along with her sleeves, her gold coif, her kirtles and bonnets, her amethyst ring, and a ring set with a diamond rose. She could stroll in and get dressed. But you cannot make a wife out of bonnets and sleeves; hold all her rings together, and you are not holding her hand.

Christophe says, ‘You are not sad, sir?’

‘No. I am not sad. I am not allowed to be. I am too useful to be sad.’

Hilary Mantel, The Mirror & the Light

What I’ve been reading

Using Choosy to open Zoom links faster on the Mac

Like many people lately, I spend half my day in Zoom meetings.

One (slightly) annoying thing about clicking on a Zoom link from Slack or an email: To enter a meeting, you first get (1) bounced to a web browser, that (2) asks if it can open an external application, that (3) you then have to click to confirm. And then, (4) you have to close the browser tab.

There are browser hacks you can do on Chrome to make the browser not ask about opening zoom, but a better solution on the Mac is to use Choosy.

Choosy is a great little app that you set as the Mac’s default browser, and it then sends links to different web browsers based on the URL. So you can set a rule to make Zoom the default browser for URLs matching Zoom meetings.

  1. Go to Preferences > Advanced, and create a new rule.
  2. Add a condition: web address matches the regular expression of ^https://.*zoom.us/j.+$

image alt Choosy preference screenshot

Note that if someone uses the /zoom command in Slack, that will bounce the browser because it has to authenticate.

Notes on The Unicorn Project by Gene Kim

The Unicorn Project Cover

Recently I listened to the audiobook of The Unicorn Project by Gene Kim. In the past, I’ve recommended its predecessor, The Phoenix Project, to multiple people, albeit reluctantly and tinged with embarrassed explanations about the format—the “business novel.” The author, Gene Kim, presents his thesis as a fictional narrative constructed solely to serve that end. The Ur-book in this genre is The Goal, which lives on more than one list of the best business books. (Both Phoenix and Unicorn specifically reference The Goal.)

I found The Unicorn Project to be slow to start but ultimately covering a lot of valuable ground. It’s most useful as a DevOps introduction for junior software engineers and for those who have only ever worked in dysfunctional organizations. It’s a great illustration of the power of continuous delivery and DevOps practices—continuous integration, developers working directly with the business, replicable builds, etc.—in a way that some people may find more powerful and digestible than a traditional exposition. Stories can be more powerful than facts.

But if you’re already aware of the concepts, you’ll spend a lot of time saying, “I know, I get it, dev environments need to be easy to set up and reproducible, I get it.” Some of the story sounds over the top—and it is—but having been on calls with big organizations trying to release software, it’s unfortunately very, very accurate.

The following are the concepts I found most useful:

  • The three horizons
  • The five ideals
  • Core and context
  • Complection

The three horizons

The three horizons is the concept that I find myself coming back to most. I am not sure how accurate it is as a top-level classification of businesses, but I have found it useful as a model.

Geoffery Moore created the three horizons to explain and categorize different types of businesses (or business units):

  • Horizon 1 is the legacy core business that generates the bulk of a company’s revenue and profitability. It’s predictable and is where process can and should be applied to drive incremental improvements and change. On the downside, it is where bureaucracy thrives and can resist needed change. It’s a gravity well that left unchecked will consume most of the company’s resources—after all, it’s where the bulk of revenue comes from. While you can reduce costs and maximize profitability in the short/medium term, any Horizon 1 business will eventually fade, since any profitable businesses invites competition that inevitably erodes profitability.
  • Horizon 2 is a smaller, emerging business that represents a possible future of the company. These businesses aren’t necessarily profitable—certainly not at first—but represent informed bets the company is making to find higher growth areas. The objective for a Horizon 2 business is to grow up into a Horizon 1 business.
  • Horizon 3 businesses are the true experiments. They require a culture of learning, and a willingness to prototype and iterate quickly to evaluate (1) market risk (does it solve a customer need), (2) technical risk (is it technically feasible), and (3) business model risk (is it a financially viable engine of growth). Many of these will fail. The most promising of these graduate into Horizion 2 efforts.

Horizon 2 and 3 are about learning-R&D, experiments, new markets, new customers, new businesses. Horizon 1 is about process and compliance. These will by nature clash.

I am by nature somewhat of a process person, so this model has been useful to remind me to try to ensure our team spends enough time in both Horizon 2 and Horizon 3. This requires (1) protecting people’s time, and (2) having a tolerance for certain teams not following established process for organizing our creation of software. But the risk of over-optimizing for the existing business is slow death.

The five ideals

You can argue the whole book is an elaborate mechanism for Kim to propose and illustrate his the five ideals for leading a technology business:

  • The First Ideal: Locality and Simplicity. In your software architecture, ideally introducing a new feature only requires changing a small amount of code in a single codebase, as opposed to many changes in many codebases. Complexity makes everything harder, and slows you down. The simpler your code, systems, and organization, the better off you will be. You don’t have locality and simplicity, making changes can be tied up in cascading, unknown dependencies. Or it requires the coordination of many different teams. This ideal directly leads to the goal of product-aligned, cross-functional teams that can deploy their own software to production.
  • The Second Ideal: Focus, Flow, and Joy. How does work feel? How it feels doesn’t matter for its own sake, but it is a leading indicator of the effectiveness and efficiency of how work gets done. Developers do their best work when they can focus on a single, difficult problem, and when their tools work with them rather than against them. Can developers work on one thing at a time, get fast feedback on their work, and quickly push their work to production? The way I think about this is that this ideal is about maximizing the amount of time a developer spends effectively thinking about and working on problems, and minimizing the amount of time lost on friction—waiting for others, waiting for builds, struggling against tooling, etc. This ideal also points to working in small batches—Theory of Constraints—both because it’s more efficient, and because it’s more enjoyable.
  • The Third Ideal: Improvement of Daily Work. In other words, continuous improvement. It’s essential to prioritize time to improve workflows and pay down technical debt, since incremental investments can pay off big over time.
  • The Fourth Ideal: Psychological Safety. Psychological safety is one of the top predictors of team performance. People need to feel safe to take risks and experiment and empowered to tell the boss something she doesn’t want to hear. A key example of this is a blameless postmortem. When there is a major customer-impacting outage, everyone should feel safe enough to raise their hand and say, “I might have caused this” and talk through tough issues with an eye for discovering the truth—not for assigning blame or ridicule. Without this, learning and improvement are impossible.
  • The Fifth Ideal: Customer Focus. Ensure that developers (and development teams) are focused outward towards the wider world, and not inward towards itself, turf, internal company concerns, or shiny technology for its own sake. A business needs to deliver value to customers to stay in business, and relentless focus on that leads to better decisions about technology and product. Ask yourself: does this actually matter to customers—i.e., are they willing to pay for it? One way to think about this is core vs. context, which I discuss below.

None of these ideas are novel to this book. But the categorization is useful for thinking about these concepts and seeing them in the world. That sounds like faint praise, but it’s not. These are essential concepts, succinctly captured.

Complection

Complect: To intertwine, or interweave, or to turn something simple into something complex.

The idea here is that software systems become complected over time, as different areas become more tightly coupled. It becomes harder (or impossible) to work on a single system because of unpredictable effects elsewhere. This is the inverse of encapsulation, although Kim uses the term “locality.”

… ‘Think of four strands of yarn that hang independently—that’s a simple system. Now take those same four strands of yarn and braid them together. Now you’ve complected them.’ Both configurations of yarn could fulfill the same engineering goal, but one is dramatically easier to change than the other. In the simple system, you can change one string independently without having to touch the others. Which is very good. However, in the complected system, when you want to make a change to one strand of yarn, you are forced to change the other three strands too. In fact, for many things you may want to do, you simply cannot, because everything is so knotted together.

Is this term useful? Maybe. I’ve always talked using words like “encapsulation” or “avoiding cross-dependencies” or “loosely-coupled” which has always gotten my points across.

Core and context

The purpose of a business is to produce a thing to sell to customers.

Core activities are those that customers are willing to pay for. Context is everything else, i.e., all of the activities that support the business but customers don’t care about and don’t create a competitive advantage—payroll systems, email, bug trackers, etc.

The point is to minimize the amount of money and effort going into context activities, and maximize the amount of time going into core. Your product should be world-class. Your email system simply needs to work. This is a great rule of thumb for deciding whether to build or buy.

One thing that The Unicorn Project does well is showing how, as time passes, what was once core can become context. Late in the book, the protagonists, forced to cut costs, struggle to decide what they should stop doing and outsource. Decades ago, they built a first-in-class ERP system, which proved a huge competitive advantage. But now there are off-the-shelf solutions that are just as good. It’s no longer a competitive advantage; it’s a distraction.

It’s important to periodically evaluate what you are working on to ensure you are spending the most time possible on investing in producing and improving the things that people actually want to buy.

Conclusion

The Unicorn Project is an interesting artifact of a book. The concepts are powerful and powerfully stated. But you have to wade through a lot of words to get to them, which I found frustrating at times. (I “read” it as an audiobook, which is probably the best way to tackle it). I think for many people, this is an ideal introduction to these DevOps concepts. The stories that a team tells about itself create a shared sense of purpose and reality—and this book can jumpstart that. You can steal these stories as a shorthand for the work that you need to do.

You sign up for things and get them done

You are about to violate a key leadership rule: “You sign up for things and get them done. Every single time.”

Leaders set the bar for what is and is not acceptable on their teams. They define this bar both overtly with the words they say, and more subtly with their actions. There are two scenarios that may play out when you’ve reached Meeting Blur: either you don’t change anything and do all of your work poorly, or you drop some of that work, which equates to a missed commitment. While the optics on both scenarios are bad, what is worse is that by choosing either course you signal to your team that these obvious bad outcomes are acceptable.

Seem harsh? Yeah, I’m a bit fired up because I think leaders vastly underestimate the impact of actions we rationalize as inconsequential. Let’s play it out once more. Thinking I am being responsible and helpful, I sign up for things. I do this repeatedly and sign up for too many things. Over time, I realize I’m overloaded, so I back out on some commitments. Where’s the flaw? Because I could not initially correctly assess how much work I could do, I’m signaling to my team that it’s okay to back out of commitments.

Michael Lopp, The Art of Leadership: Small Things, Done Well

Following through on personal commitments, 100%, all the time, is an aspect of leadership I try to hammer into any manager on my team. It’s also the one I’m most paranoid about falling short on myself.

Honoring and extracting reality

Another thing I learned along the way is how critical it is to have senior leadership support. And support in actions, not words. Senior leaders need to demonstrate their commitment to creating a learning organization. I will share the behaviors I try to model with my teams. I believe passionately in honoring and extracting reality. If I am a senior leader and my team doesn’t feel comfortable sharing risks, then I will never truly know reality. And, if I’m not genuinely curious and only show up when there’s a failure, then I am failing as a senior leader.

Courtney Kissler, VP Digital Platform Engineeering, Nike. From the forward of Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations.