Inefficiencies of a software engineer, reading stories and writing things down

The inefficiencies of software development

Developer efficiency is a hot topic, and it's a great conversation starter amongst executives. But at the risk of generalising, it starts a conversation that doesn’t yield much beyond time sheets and work from the office. And that’s because developer efficiency focuses on an engineer and their time, inputs to a complex system that produces software. You can make any optimisation to these inputs, but not knowing what slows down product engineering leads to vigorous tinkering and percussive maintenance; you end up discombobulating a team into further disarray.

Before you pick up your mighty mace or your favourite ad hominem and tear me to bits, I must say that good, efficient developers are indeed the bedrock of any company that produces high quality working products at pace. Table stakes if you may. But most workplaces, if not all, exact a certain tax that even highly efficient unicorns, by that I mean software engineers, have to pay. And that makes them less efficient. And even if they are efficient in their small little silos and IDEs, their teams suffer incessantly. Let's call this a workplace tax for now. What are the constituents of this workplace tax? This is what I wanted to catalogue, dear reader, from my own experiences and ones I have observed around me. If you are bothered about developer efficiency at your workplace, you should understand this tax too and learn how to reduce it.

In this and the next few letters, I will try to expound on a major source of inefficiency in the product development process. I won’t attempt problem-solving or thought leadership here, I am not smart enough to provide generic solutions to very specific problems, but I will happily correspond with you over email or call to discuss and help you arrive at a solution.

We start with org structure, the most neglected yet the most taxing of the inefficiencies, something that’s never in the hands of an engineer and yet harms her the most.

Org Structure

Photo by Edvard Alexander Rølvaag on Unsplash

Excessive Hierarchy: The Burden of Too Many Approval Layers

One of the most common pitfalls in large organisations is the presence of excessive hierarchy. This manifests as multiple layers of management, each requiring approval for decisions, no matter how minor. While some level of oversight is necessary, too much can lead to:

1. Decision Paralysis: Simple choices become bogged down in a quagmire of approvals, slowing down the entire development process. Engineers may find themselves waiting days or even weeks for approvals on minor technical decisions, leading to frustration and delays.

2. Demotivation: Engineers often feel disempowered when they can't make decisions about their own work. This lack of autonomy can lead to decreased initiative, reduced job satisfaction, and a decline in innovative thinking.

3. Increased Time-to-Market: The accumulation of delays at each approval stage can significantly extend project timelines. In fast-moving markets, this can mean the difference between being an industry leader and falling behind competitors.

4. Risk Aversion: With multiple layers of approval, there's a tendency for each layer to avoid risk, leading to overly conservative decision-making. This can stifle innovation and prevent teams from exploring potentially game-changing ideas.

Siloed Teams: The Pitfall of Isolated Expertise

Another significant obstacle to efficiency is the existence of siloed teams. This occurs when different departments or units within the organisation operate in isolation, with limited interaction or knowledge sharing. In software engineering, this can lead to:

1. Duplicated Efforts: Teams may unknowingly work on similar solutions, wasting valuable time and resources. Without visibility into other teams' work, engineers might spend weeks solving a problem that another team has already addressed.

2. Inconsistent User Experiences: Different teams may develop conflicting interfaces or functionalities, leading to a disjointed and confusing product for end-users. This lack of cohesion can damage the overall quality and usability of the software.

3. Missed Opportunities for Innovation: Cross-pollination of ideas between different specialties often leads to breakthroughs. When teams are siloed, these opportunities for creative problem-solving and innovation are lost.

4. Knowledge Hoarding: In siloed environments, teams may develop a sense of ownership over their knowledge and be reluctant to share it. This can lead to inefficiencies when other teams need to reinvent the wheel or struggle with problems that have already been solved elsewhere in the organisation.

Misaligned Incentives: When Teams Pull in Different Directions

Misaligned incentives occur when different teams or departments within an organisation have conflicting goals or performance metrics. This misalignment can create friction and inefficiency in software development:

1. Conflicting Priorities: For example, if the security team is incentivized to minimise risks while the development team is pushed to release features quickly, it can lead to constant tension and delays in product delivery.

2. Suboptimal Solutions or local optima: Teams may prioritise their own goals over what's best for the product or company as a whole. This can result in features or architectural decisions that serve a specific team's metrics but detract from the overall product quality or user experience.

3. Decreased Collaboration: Teams may be less willing to help each other if it doesn't align with their own incentives. This can lead to a culture of "not my problem," where issues falling between team boundaries are neglected.

4. Short-Term Thinking: Misaligned incentives often encourage short-term thinking to meet immediate goals, at the expense of long-term product health and sustainability.

Unclear Roles and Responsibilities: The Confusion of Blurred Boundaries

When roles and responsibilities within a software engineering team are poorly defined, it can lead to significant inefficiencies:

1. Role Overlap or Gaps: Important tasks may be duplicated, wasting time, or worse, fall through the cracks entirely. This can lead to critical components being neglected or unnecessary conflicts over ownership.

2. Decision-Making Uncertainty: Team members may be unsure who has the authority to make certain decisions, leading to delays or conflicts. This uncertainty can paralyse teams, especially in critical moments when quick decisions are needed.

3. Inefficient Resource Allocation: Without clear roles, it's challenging to assign the right people to the right tasks. This can result in suboptimal use of team skills and expertise, reducing overall productivity.

4. Accountability Issues: When responsibilities are unclear, it becomes difficult to hold individuals or teams accountable for outcomes. This can lead to a lack of ownership and a tendency to shift blame when problems arise.

Poor Communication Channels: When Information Doesn't Flow

Effective communication is the lifeblood of any successful software engineering team. Poor communication of goals and processes can severely hamper efficiency:

1. Information Silos: Critical information may not reach all relevant team members. This can lead to decisions being made without full context, potentially resulting in errors or inefficiencies and significant time wastage in rework.

2. Misunderstandings: Lack of clear written down communication can lead to misinterpretation of requirements or expectations. In software development, these misunderstandings can compound over time, leading to significant deviations from intended outcomes.

3. Reduced Collaboration: Without effective channels, spontaneous collaboration and idea-sharing are less likely to occur. This can stifle creativity and problem-solving, as team members miss out on valuable insights from their colleagues.

4. Delayed Feedback Loops: Lack of focus on communication and feedback can slow down implementation cycles, whether it's between team members, with stakeholders, or from users. This delay can result in teams working with outdated information or continuing down problematic paths for too long.

The organisational structure of a company can have profound effects on the efficiency of its software engineering teams. Issues such as excessive hierarchy, siloed teams, misaligned incentives, unclear roles, and poor communication channels can significantly impede team performance, innovation, and overall product quality. Recognising these structural inefficiencies is the first step towards creating an environment where software engineering teams can thrive and drive the organisation's success.

(next letter would continue on this theme and talk about processes, team sizes and more. if you don’t want to miss out, hit that subscribe button)

---

Reading Stories

I read Hyperion by Dan Simmons this month. Simmons is a very interesting person, but that’s a story for another day. Hyperion is a remarkable book and all the time while I was reading the book, I was kicking myself for not picking it up earlier. I mean this book was published in 1989 and I am reading it 35 years too late! Apart from Simmons I also went into digressions like the poem by Keats, the Titans and many other things I might bring up later.

A modern day Canterbury Tales, the story has many timelines, some interlocking, all leading to a present day conundrum- what is the fate of humans with the advent of AGI. Simmons presents his perspective in a delicious storybook nobody should miss. It is craft, creativity, story telling of the highest quality.

The book has an immediate hook, I wasn’t expecting that. And there are 6 narrators who are going on a pilgrimage that will lead to the death of others. The genres or subjects Simmons brings in from historical fiction, detective noir, drama, comic relief, it’s not easy to put all this in a book, and it is all immensely enjoyable. While I enjoyed reading all the chapters, I especially liked the Priest’s story, reminded me of reading Shogun all those years back, fascinated about how missionaries over centuries ago travelled to seek knowledge or just roam about, without any idea of culture or language of the place they were visiting. These days we spend more time researching where we are going than the time we spend there - we have become so fragile. Also liked the Detective’s tale, its where Simmons starts picking up speed on the science fiction, and the Consuls story that really makes you think about what it is to have a home and family. Anyway, don’t want to spoil the book for you. Go read it if you haven’t already.

If you have a book recommendation for me, please do comment or email me. You can reach me on goodreads as well.

---

On Writing things down, on paper

Starting as a hobby long time ago, fountain pens, inks and paper have become my primary mode of writing. I still write a lot digitally, what with work and emails. But what I have learnt to love and respect is writing stuff down on paper. When I (re)started around 5 years back, I was surprised by the almost mystical sensation of finding clarity while writing. Over time I have realised that the benefits of writing things down with hand on paper has some distinct advantages. Writing takes time, while writing I am focused on what I am writing about and other things do not matter, there are no notifications on paper that I have to look at. Writing hypotheses, context, thought processes, all of this allows me to have a dialogue with myself, most importantly it has given me a lot of time to think about something before responding. Writing also is therapeutic, I open my thoughts wholeheartedly to, well myself, without hiding them in some corner and shutting them down. The tactile feeling of writing with a fountain pen, the colours of ink I use and how they look on paper, all of these combine together to make wholesome experience. 

While recording thoughts and notes digitally has its advantages (see Roam Research/logseq/obsidian/excalidraw), writing on paper allows for a more fluid visualisation experience that cannot easily be replicated on a computer right now. The quickest way to transfer what you think to something tangible happens with pen and paper, or on a whiteboard. I know in some years, this gap will vanish, but for now, I draw (see the cover photo of this letter) and write without a worry of judgement, and I am just not able to do that digitally. Either that or it takes so much time for me to draw something on the computer/ipad that I get distracted.

Over years I have gone down many (costly) rabbit holes about pens and nibs, paper and inks, paper sizes and generally being a stationary hoarder. That meaningless exercise has now become a really important part of my learning process. I wholeheartedly recommend writing down on paper to anyone who is trying to find clarity in situations. It doesn’t matter which process you follow, of which framework you use. Most of the things that I write here are written on paper first, and then edited.

What is your process like ?

(the pen in the picture above is a Ranga Model 4CS, an Indian handmade ebonite pen. its nib is made of titanium using an archaic butterfly nib-tipping process)

---

Currently Reading

1. Nexus: A Brief History of Information Networks from the Stone Age to AI by Yuval Harari

2. Jade City by Fonda Lee

3. Another chapter in Lee Child's Jack Reacher books