The invisible skill at the heart of every technologist

There’s a skill that’s core to success at all levels of software development, but it hides like dark matter: inferred, rather than observed, lurking beyond description and discussion. On a gut level, we know it’s necessary to the work, but we don’t often know how to teach it. You’ve either got it or you don’t in the perception of the average workplace.

Let’s call this skill investigative reasoning. Or, more simply: “working the problem,” as Apollo legend Gene Kranz might frame it.

(Let me know if MIT press has a $70 textbook explaining this, situating it in computing or engineering, and giving it a name.)

This pile of skills manifests as a reflex to make sense of an issue or failure, locate its causes, and design a fix. Without these abilities, you’ll be helpless in the stew of complex abstractions that define modern computing.

Consider a common software development workflow:

• You write some code
• It’s compiled, bundled, or otherwise transformed
• The transformed code is transferred to a host device
• The host runs the code

Failures can emerge at or after any of these steps. The code may have errors which prevent compilation. The code may have errors despite compiling fine, and they manifest only at runtime, during certain circumstances. There may be a failure in the systems that package and transfer the code, preventing it from running.

The experienced practitioner understands their job is to analyze the problem, develop a hypothesis for its source within the many layers of technology they’re using, and validate the hypothesis by checking logs, adding print statements, starting an interactive debugging session, or using other means of peering inside of the machine. Working the problem may require research: finding online discussions of similar issues, or consulting documentation.

But core to this activity for both the novice and the expert is a simple conviction: problems can be understood and investigation can yield solutions.

At any level, hobbyist to advanced professional, confidence in this premise is the indispensable requirement. Without the confidence to work a problem, it’s unlikely anyone can make progress in building and integrating technical systems.

Domain experience helps to work a problem. Every class of system, from microcontrollers to native applications to web apps, has its own set of quirks, its own unique abstractions and design decisions where trouble may sneak in. But working the problem is a skill that transcends domains. Fixing a broken software development toolchain can prepare you for troubleshooting the boiler in your basement.

The paranoia in hiring for software roles, I’m certain, is rooted in the fear that the organization will hire someone who is unable to work a problem, outsourcing their investigative reasoning tasks to others, depleting productivity per unit of headcount budget.

Working the problem is also a feature of seniority. Translating business requirements into software development tasks is an expression of this skill, as is understanding how technical tradeoffs (debt) in existing systems will impact the needs of new features. The more senior a practitioner is, the more likely they are to feel a sense of confidence working a problem no matter how many layers stack up—even when those layers are “non-technical,” derived from business or customer needs.

We laugh at the fixation on languages or frameworks in job postings, but even the most thoughtful organizations struggle to identify this set of activities as a job requirement, much less map their outcomes within the team’s problem space.

Not every person is the ideal practitioner to work a given problem. The startup costs in research, learning new tools, or picking up domain-specific expertise, may outstrip any return on investment. Still, I have to think that applying a more conscious growth mindset around this would give our industry better results in recruiting, professional development and retention than we have today.

How can we elevate our understanding of these skills? How can we instill the confidence needed to work a given problem? How can we better train people to be able investigators?

I think there’s a lot of missed opportunity here.