A recurring question in software engineering leadership is: how much process does your engineering team actually need?

I formally studied software engineering with the explicit goal of developing engineering leadership capabilities and systematic problem solving. I also have been gathering knowledge in the field in different cultural contexts (Tunisia, France, Germany, USA) and domains (industrial design & PLM, automotive, mobility, AI systems, cloud-native systems and personal finance). That diverse experience helped me to observe and identify two recurrent biases software engineers should be wary of:

1) The strong belief that “good software engineering is an art”. I have always seen in the word “art” an attempt to obscure something, because by definition, art is not objective, measurable, reproducible or comparable. Art is emotional, inexplicable, gut-feeling-driven - almost the perfect opposite of engineering, but that’s a different debate altogether. I started with “strong belief” very deliberately: The creative emotions and feelings around writing code, while justified and a delightful part of the job, may devolve into viewing software engineering as something shapeless, formless, subjective, magical almost, non-reproducible through any kind of learning, empirical study or research, instead only reachable by sheer ambiguous “experience”, whether that experience is actually building expertise or consists of doing the same thing for 20 years. This is obviously an issue.

And it is, to be blunt, often either:

• a beginner’s understanding, because of the lack of real breadth or depth of experience, that makes experience sample sizes small enough for a beginner to draw hasty conclusions that are often wrong - and I’ve been a beginner, and done just that. When you haven’t seen much of something and haven’t done much of it, you may get the idea that it is easy to pull it off on gut feeling only because it has always worked for you so far, forgetting that you probably have only been solving small simple problems that have been solved countless times by your peers and seniors in the industry, and mistaking them for “software engineering” as a whole.
• the attitude of someone who may be experienced, but who never really had to work on highly constrained, highly complex projects (scale, number of stakeholders, number of developers, etc.). It almost becomes a desire to belittle and systematically dismiss any effort to standardize, quantify, study or formalize software engineering practices, preferring “flexibility” and “creativity” which are really code words for not wanting to do the work or learn from others’ experiences. The packaging of this attitude varies, but I have learned to spot it by watching for overconfident statements like:
  • “my favorite language/stack can do everything” (which it may well be able to, but the person speaking may actually not be able to deliver that)
  • “I don’t use a debugger, print statements and a cup of coffee” (and a few hours that you could have saved with a few conditional breakpoints)
  • “I feel this is as optimized as it can be” (well, except you haven’t profiled it or even run a static code analyzer on it so you don’t really know, you just know that it is as optimized as you can make it, which is quite different)

I have seen this in so many teams and with so many projects, that now anytime a developer (especially a senior who hasn’t touched anything but one technology/stack for a long time) claims they are “a code craftsman”, “a software artisan”, or that “X is the best technology” etc. a red flag goes up in my mind. Anytime I read “method/technology X is dead”; my skepticism activates - for the simple fact that that doesn’t mean much, really, we don’t even agree on what constitutes the death of a technology. Using that label to justify one’s choices as a developer, hiding behind a certain mystique, usually demonstrates a lack of accountability or understanding of what software is, how it should be built, and the why behind the how.

“But Yazid, you present yourself as someone who is overly philosophic about software engineering, so why can’t others be allowed to do it?” - great question. Do it, please, I deeply think that debate and conversations in our industry help all of us get better and understand things better. The line I draw is: I’m always open to my mind being changed, or to being convinced with solid arguments of a strategy, a method or a solution. The “vagueness artisans” I talk about here are rarely open to that, and a definitive way to spot them in my experience is to challenge their general and overconfident statements: their reaction will tell you, if they listen and go back and forth reasonably, you’re in front of a good artisan, one that knows their stuff and doesn’t hide behind ambiguous authority, open to learning and discovering things they haven’t considered. If not, you are probably in front of a “vagueness artisan”: someone who justifies their stubbornness with vague references to a questionable artistry that only they can understand.

Anytime I see someone with narrow experience (if not a literally single depth-first search branch) tell the world their theories about software development, all of software development, any software development, and dismiss everything everyone else is doing, I know I’m dealing with an “vagueness artisan”, not a software engineer.

Don’t get me wrong, an “vagueness artisan” will do some things well, they could write beautiful well-structured code, they could build systems that work as envisioned by their clients, but most of the time they’ll take a lot longer to do so than if they sought actual lessons learned from peers and literature. You can figure out the SOLID principles on your own, they may help you if the problem setup is right, but you could conversely reach that end point much faster if you just read them, saw some examples and implemented them instead of squeezing them out of your brain after a handful of projects where you built broken database schemas.

I am, however, by no means advocating for firing all of your artisans. They do add value and they do have a place within teams provided they do not disrupt the process or the flow of things too much especially if they’re very vocal and critical of everything and everyone else. If you have an artisan on your team, provide them with enough space to be creative and to be to do things as they like them to be done. Just make sure they understand what is expected of them, quality constraints, sticking to the requirements and standards etc. Sometimes, that would mean giving them a piece of the software to build that is highly decoupled from all other pieces and does not require intense collaboration with other colleagues, except via very clearly defined interfaces. “I love coding but I hate working with people” is not a rare statement from software developers, and accommodating someone who can work independently and autonomously to deliver good results may be the way to go there. You would be surprised what some artisans can deliver in that setting. That is where many artisans (true ones) actually shine and provide high value.

2) The strong belief that “without (my favorite) formalized, explicit, heavy, tracked process we’re doomed”. “If you’re not doing OKRs you’ll be out of business”, “if you’re not doing CI/CD you’ll be out of business”, etc. I realize I may have sounded like this type in the previous paragraph. Notice however how I started the description again with “the strong belief”, it has to be a belief and it has to be strong. “A belief” in this context is an unjustified strong conviction that is more based in irrational/subjective thought (one that probably goes like “the thing I don’t know can’t possibly be better than the thing I know”) than in actual study and learning.

I’m sure you have encountered this multiple times throughout your career where someone would be a zealot for a certain methodology and can never see the criticism that people may raise against it. Their conviction is oftentimes so strong that it does not accept any criticism or discussion of the methodology they so strongly believe in, and that’s where the qualifier “strong” comes from. It is OK to adopt a certain methodology that works for most of your use cases and to reflect upon its advantages over other methodologies and how it’s making your process more streamlined and your coding easier. Where most developers go wrong is trying to generalize a methodology that they know or like as being the best over “software engineering” in general, flattening all of its facets and the diverse problems it solves, exactly just because they like. Extrapolation must be exercised with extreme care, and understanding that sometimes the simple “let’s eyeball it” is enough for most PoCs (Proofs of Concept) and small enough tasks/projects is key. You don’t need a full project structure and process to produce a simple PoC the goal of which is to dip your team’s toes in a new technology or a new way of doing things.

I have however to recognize that while “vagueness artisans” are by definition independent individuals who dislike the very concept of organized/structured work, pedantic cold robots are very likely to be drivers of or reflect corporate rigidity. In that sense, you are more likely to encounter cold robots, whether minimalist (I think my process is the best) or overbearing (everyone must follow my process), which makes the comparison somewhat unbalanced.

But ultimately, “vagueness artisans” and robots are two sides of the same coin: overconfident, unable/unwilling to justify or reason or accept alternatives to their choices or opinions, both scared of the key question “why?”, because that is the question that unravels their badly constructed stubborn beliefs.

How does this fit with the picture of a startup?

Startups are known for the funding-scarcity-driven “quick and dirty” approach, “move fast and break things”, “fail early fail fast”, which were funny enough adages pulled out of the context where they make sense (Software QA) and generalized into pitch decks for VCs. In the business sense, there is some meaning to this. However, using that as a reason to not adopt any process or structure is often a mistake startups pay a high price for later, since software issues compound over time.

I always come back to my favorite quote from the movie “Body of Lies”, stated by the brilliant Mark Strong as Hani Salaam, which is “Urgency does not call for changing methods that work, for methods that do not work.”.

Indeed, many times the feeling of urgency would have you chasing the target the wrong way, and the imperative to deliver on those work items urgently would have you going down the wrong path for so long that you lose sight of the actual goals. Ironically, moving fast and breaking things can turn into delaying a realization deliberately because of not questioning the current path. Stubbornly prioritizing building and showing something for it over thinking of what you’re building and how it is relevant to your goals is a mistake I’ve seen done so many times, from large companies to startups. A certain amount of failure and learning is acceptable of course, that is the very core of any engineering approach. The problem arises when the imperative to deliver has teams skipping over the very important phase of thinking of the “why” of it all and the “how” of it all, as related to the “why”. In other terms: skipping over things like requirements engineering and architecture design and jumping right into coding a vaguely understood solution.

I draw the line at PoCs. PoCs don’t need as much forethought as full projects or products, they often just aim to prove a few possibilities or a statement of feasibility. In that sense, quick and dirty is the way to go, with the understanding (oh so elusive still) that the outcome is not a working product and must not just be pushed to production.

So how much process does my engineering team need?

Anything beyond a PoC needs some process. “Some process” is vague enough, but the rule of thumb is: if you and your team still don’t have a good understanding/consensus on the “why” or the “how” of it all, there’s a lack in your process. It doesn’t mean “add forms, spreadsheets and meetings and it will fix itself”. It means sit down with your team, identify ambiguities, add activities in your pipeline that aim to effectively and systematically lift the ambiguity. It also helps to regularly investigate why these ambiguities have still managed to slip through. With many teams I worked with, I always encouraged honest and direct postmortem reports that aim mainly to identify the root causes and remediate their effects.

For example, in my two most recent experiences with teams developing web and mobile apps, I initially set up simple Kanban processes that gradually evolved into a more streamlined Scrum approach, where prioritization and clarity of stories and tasks took precedence over aspects like estimation. The cost of bad prioritization and bad understanding or translation of requirements into “developer tasks” has proven much higher than the one of “bad estimates”. Quite often, bad estimates originate from a bad understanding of requirements.

In conclusion: it always comes down to honesty with oneself and with others: coding is an art form in the choices you make as a developer and your ability to adapt, invent and innovate, however that needs to be backed by humility and the willingness to learn from what others did before. It may not (or certainly will not, rather) apply to the problem you’re trying to solve exactly like it did to their problems, but it can provide you enough guidance to avoid wasting time reinventing the wheel. It is a good exercise, reinventing the wheel, but it stops being good when it holds back an entire team as you egoistically waste time they need you to use to deliver.