Topic 7 Posts

explainer

Dev Diaries: storytelling builds a core of community experts

Stellaris is a strategy game where players can build the stellar empire of their dreams. By navigating a series of interlocking economic, scientific and political systems, players create a unique storyline of exploration and contact with dozens of species.

On a spectrum of simplicity versus complexity, this is at the far opposite end from checkers. Still, once you learn the basics of managing your empire, Stellaris can be a blast.

The business model for Stellaris is echoed in many strategy titles. The 1.0 version of the game forms a foundation upon which years of new functionality and paid DLC can be designed and shipped. In the case of Stellaris, 20 expansions are available, ranging in price from $8 to $20, with the game itself retailing for $40. These expansions add everything from new species to entirely new storylines, political systems, and technologies. In Overlord, you can create new stellar megastructures and better specialize vassal states, while Synthetic Dawn introduces AI civilizations and hive minds.

This strategy makes the lifetime value of a committed Stellaris fan as high as $280 over the course of six years of releases, and there are further expansions on the way.

The approach of adding both new content and new game systems keeps things fresh, engaging both experienced players and newbies alike. It allows amortization of initial development expenses over a longer period, while maintaining the viability of the base game’s price.

Of particular note is that every major DLC release coincides with an update to the game that every customer gets even if they don’t pay for any expansions. The DLC model makes it economically viable to continue maintaining the intricate systems that drive the game, fixing bugs and eliminating tedium.

Stellaris is a better game today than it was six years ago.

How do I know all this? It’s easy.

Occasionally, I read the developer diaries.

Software storytelling

Software is itself a strategy game. We are constantly pivoting between the scope of our ambitions, the time cost of building systems, the complexity cost of systems already in motion, and the coordination and labor costs of the humans driving all of it. None of this is easy, and much of it is hidden well beneath the surface, out of view of most software consumers.

Indeed, it is a frequent gripe of game developers in particular that players rarely understand just how expensive it is to build a game to any high standard.

In the case of Stellaris, the business model requires at least a handful of engaged players to grasp these tradeoffs, and the game’s underlying systems, well enough to have a conversation about making changes and fixes.

Only by having informed dialogue with the community could Paradox Development Studio hope to learn what they need to make the game have lasting business value six years running. A game that no one wants to play isn't a viable commercial enterprise.

Their solution? Like many indie games, Stellaris publishes developer diaries. These represent an exhaustive body of regular dispatches from the code mines where the game is constantly reborn, revealing everything from the high level goals of a new release to the underlying minutiae of the development process. As of this week, Stellaris devs have written 279 dispatches, posted to their forum, where fans of the game can respond.

Dev diaries provide a tantalizing preview of upcoming adjustments to the game, which builds interest and even stimulates participation in betas and previews, when necessary. The diaries describe changes, fixes and replacements to game mechanics, show off new UI elements, and explain the rationale for changes and investments.

While only a small proportion of the game’s player base reads and engages with the diaries, the ones who do understand both the developers themselves, and their output, on a level that’s rarely seen in the software business.

This opens the possibility for communication and feedback with much deeper shared context between the most invested community members and the development team. As changes are in-flight, discussion in response to dev diaries allows the team to gauge sentiment and interest, while players can ask questions and learn more about goals and hiccups.

The result is a fruitful venue for dialogue and a community that can be as informed as it wants to be. This matters in a multiplayer strategy game. You need people eager and willing to participate, or there will be no one for new customers to play with.

Broader community lessons for all projects

“Community” is a frustrating, poorly understood concept in technology spaces. It’s not stickers or t-shirt guns. It’s not a thing you bolt onto a conference. It’s not a thing you can buy.

Community is the sum total of people who are invested in the success and fruitful application of a project, as well as their bonds with one another. Community emerges around a project when that project’s values and ambitions are shared by a larger group than just its developers. Community could be measured in the volume of people who would have a genuinely shitty afternoon if they learned your project was dead, as well as the resulting acts of commiseration between them.

In the 60’s, when it seemed Star Trek might die, letter writing campaigns to NBC bought the show extra life. After the show’s cancelation, the community of people served by Star Trek’s vision of the future lived on, organizing spontaneous conferences to discuss the show, and eventually, to meet its stars and creators.

Not everyone is making Star Trek or Stellaris. But if you want your project to punch above its weight, you need to build the ranks of people willing to work for its success.

Developer diaries are a powerful lever on this point. They create a regular cadence for project storytelling, they provide context around your decisions, they invite people to engage you with ideas and feedback—some of which will be grounded in the exact constraints your diaries have explained!

This creates a basis for relationships. Not just between you and the people who you hope will care about your project, but between those people as well. It’s an opportunity to create a lingua franca of the project, creating bonds between those who are similarly excited by its progress and intricacies.

Without concerted storytelling effort, the work of building software is invisible, even in the most open source of contexts. Your commits and files can only tell part of the story. Sometimes the most interesting code is the stuff you chose not to write. Developer diaries are an opportunity to reveal those decisions.

If you want a base of truly invested community members, literate in the larger philosophical and strategic underpinnings of your work, it’s worth exploring how a developer diary practice could serve your project.

Grasping the true scale of inequality

There's a problem with understanding inequality at the modern scale.

Our minds struggle to make sense of a "billion" things. Much less tens or hundreds of billions of things. Our minds further struggle to compare how a billion of this might compare to a million of that.

As a result, the everyday person has no idea just how much more money the wealthy have:

The average American believes that the richest fifth own 59% of the wealth and that the bottom 40% own 9%. The reality is strikingly different. The top 20% of US households own more than 84% of the wealth, and the bottom 40% combine for a paltry 0.3%. The Walton family, for example, has more wealth than 42% of American families combined.

This viral video tries to visualize the drama:

There have been more attempts to make the differences tangible. For example, you've probably seen the viral TikTokker who quantified extreme wealth with grains of rice. For example, if a single grain is worth $100,000, Jeff Bezos has 58 pounds of the stuff.

I'd like to offer an alternative to these laudable approaches to solve this difficult problem.

Inequality is foremost a matter of time

Work is a trade: time and energy for some amount of money. We give up irreplaceable time in our lives to pursue the goals of someone else. In exchange, we get enough money to, we hope, pay for our basic necessities: food, shelter, clothing, medical care. If we're very lucky, we make more than we need, and can use the rest for comforts and saving.

In the United States, one year of work yields $70,784 in the median case.

Aside: median vs average

As a refresher, the median value in a set of numbers describes where the middle is. In other words, there are as many values that come before as after. Medians can be helpful in statistics around inequality because they prevent extreme values at either edge from disorting the picture.

Median annual income as the unit of time-for-work

So at the middle of the pack, $71k isn't quite prosperous, but it is enough to rent a one bedroom apartment in every US state.

Like grains of rice, we can use this number to slice up inequality into numerical scales we can actually understand. So for every $71k you have stored up, that's a year of self-determination or leisure time available to you. A year buffering you from poverty and desperation.

Time leverage by annual compensation

To start, let's look at annual compensation as it yields a unit of median US income. In other words, how much does a year of work buy you in terms of the power not to work if you don't want to?

Elon Musk is doing pretty well: in just a year he was paid 142 millennia of median income. In other words, Elon made enough leisure money for 23x the duration of all human civilization.

Musk is an outlier, certainly, but you can find plenty of other dramatic examples.

Tim Cook made enough in a year for 12 millennia of leisure, and Sundar Pichai got enough for 40. Dave Clark has 800 years of leisure time at his disposal, while Satya Nadella gets 700.

Parasitizing the American healthcare system isn't a bad gig, either. CVS CEO Karen Lynch has almost three centuries of leisure coming her way, as does UnitedHealth CEO Andrew Witty. This makes sense: people will do anything to keep themselves and their loved ones alive. It's a profitable protection racket.

Everyday workers, meanwhile, earn less. Walmart, Amazon and McDonald's workers' median incomes are less than half of the national median, while Apple's is 80%.

As Starbucks workers fight, often successfully, to build a union, it's worth noting that the CEO there gets almost three centuries of leisure, while the typical barista scrapes by with only half the national median income.

Dave Clark gets eight centuries of leisure in a year while half his workers don't even get a single year. In fact, it's worse than that, because working in an Amazon warehouse can cost future earnings, due to injuries and fatalities.

I'm comfortable with the argument that being an executive of a public company takes certain specialized skills not everyone has, and therefore is due certain additional rewards. But centuries of leisure potential every year? While the typical worker doesn't even hit the median income, much less stack up any extra? That's taking so much and leaving so little.

Time leverage by wealth

But annual compensation inequality is nothing compared to wealth inequality.

During the 2016 campaign, Trump argued that he received a "small loan" of $1 million from his father, and that he'd worked hard for his wealth. Let's take the claim at face value, ignoring all his other generational advantages. That's 65 years of income, or more than an entire lifetime. Imagine what you could build with an entire lifetime of income loaned to you at the beginning of your career.

Meanwhile, back to Musk. He has 2.7 million years of buffer time stored up. That's more time than has passed since the first humans evolved. Bezos, Buffet and Gates each have around 1.5 million years.

If it sometimes seems as though Nancy Pelosi—disdainful as she is of progressive policy goals—is out of touch with the typical American, it's worth noting she has as much as 15 centuries of cash. That's enough money to last from the fall of Rome until today. McConnell isn't doing too badly, either, with 500 years stashed away. Joe Biden could retire, meanwhile, for over a century on his current haul.

AOC, it should be noted, may still be in the red thanks to student loans. She's much closer, therefore, to the typical American, which has just two years of cash buffer. It's harder for those who didn't attend high school: they have almost no buffer, at median net worth of $20k. Having a college degree, meanwhile, brings the median to four years of buffer.

This doesn't even touch the mechanics of financialization, like stock buybacks. Apple has transferred 132,000 lifetimes of wealth in this form, or 5.9 million years of the median US income.

Inhuman leverage

So we have some people, in American society, with almost no buffer at all.

Meanwhile, some among us have so much excess power in the form of time that they'll die centuries before they could come anywhere close to using all of it. This is disorting our world in dramatic ways, between the chaos of the Twitter acquisition to the fallout of Citizens United, to the ongoing consolidation of essential services.

As a whole, we're able to produce so much wealth. Does it really make sense for the most powerful among us to be so gluttinous with the rewards? Does Elon need 61,000 lifetimes of wealth? Do Bezos, Buffet and Gates need 30,000 lifetimes of wealth?

What would happen if they shared the pie a little more? How would the resulting tax revenues improve our communities through infrastructure and education spending? How would everyday lives improve with less stress, more leisure time, more time with our loved ones?

It's helpful for the wealthy that the numbers are so incomprehensibly big it takes a whole spreadsheet just to begin the conversation.

When your salary requires you not understand the labor movement

I’ve been reading Daring Fireball for something like 18 years now. I appreciate John Gruber’s insights on Apple, and find him more right than not in analyzing their products, strategy and motivations. Hell, I survived a layoff in 2020 by buying an ad on his site.

But I’ve been scratching my head at this recent remark about union drives at Apple’s retail operation:

This public enthusiasm for labor unions is manifesting in high-profile unionization drives at big companies like Starbucks, Amazon, and now Apple.

This is a strange logical construction to me, but it mirrors a larger challenge I find among pundits in understanding the current moment and movement in labor.

In one of my favorite quotes of all time, noted 20th century troublemaker Upton Sinclair wrote “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!”

The most insightful people in the game are struggling to make sense of a resurgent labor movement. But it’s not that hard to follow—if your incentives aren’t too bound up in the interests of the people who already have a lot of money.

Trouble is, that’s a hard line to walk while getting paid to write. I'm sympathetic—and unaffected. Maybe I can help.

Unions aren’t forming because they’re popular; they’re popular because they’ve become urgently needed and they’re forming for the same reason

In most people’s interactions with a workplace, the company takes too much and gives too little. The only recourse for labor is to form structures of counter-power to try and balance the equation.

You can stop reading there. All I’m going to do next is prove the point several ways, but if you came here to understand why unions are both forming and popular, you’re good to go.

CEOs, as agents of Wall Street and other financial interests, are paid hundreds of times what their workers make every year. In Apple’s case, Tim Cook took home $100m in 2021 alone. The typical Apple Store employee, making $22 an hour, would need to work 2,367 years to match Tim’s compensation.

This isn’t unusual to Apple, though. CEO pay is at an all-time high, but that’s not even the worst part. When workers create profits for corporations, what doesn’t go to the CEO is too often sucked up by shareholders in the form of stock buybacks.

Supporters of the status quo will argue that guys like Tim Cook create outsized value for companies, and deserve outsized compensation as a result. I can accept that Cook is a uniquely talented person with unique insights. Gil Amelio, Michael Spindler and John Sculley are proof enough that not everyone is suited to run Apple.

Nevertheless, I struggle with the idea that Cook deserves that much more of the pie than the people who make it possible for him to move the vast quantities of hardware and services that allow Apple to post its billions in quarterly profits.

This isn’t an argument in the abstract, either. It’s becoming harder and harder to afford the basics of life—housing, food, transportation, childcare—in the United States, precisely because of this inequality. For example:

The people with money are living the high life while wage workers are struggling to get by. But this is about more than money. Employees of large corporations are separated from decision makers by enormous gulfs of reporting structure and policy, with limited say in their day-to-day work.

Apple’s workers don’t just want more money, they want things like better scheduling and career advancement. The timing of when you work is everything: it impacts your ability to rest, to be with friends and loved ones, to meet educational goals, and otherwise determine the course of your life.

Scheduling in a recurring theme in many recent retail labor disputes, as in the case of Starbucks.

Amazon presents perhaps the most extreme example of how precarious today’s workers are. Six warehouse workers died when a tornado struck a distribution center in Illinois last year. Desperate drivers with no slack in their schedules have to piss in a bottle to meet their delivery quotas, as the company admitted to lawmakers. The company’s idea of worker well being is, in a bit that would go too far even for Severance, a phone booth-sized cubicle where workers can watch mindfulness propaganda.

Self-determination is an issue for wage earners across many sectors. The US sits on a knife’s edge as rail workers—over-scheduled and fighting for the basic right to do things like visit the doctor once in awhile—contemplate a nationwide strike that would grind logistics infrastructure to a halt. Those guys, at least, have a union.

To recap, workers are struggling with:

  • The basics of reliable scheduling and paid time off
  • Soaring costs of the essentials
  • Their ability to advance their careers
  • All the surplus value they create going to CEOs and Wall Street

In an economy that has produced enormous gains over the last decade, all of the fruits are going to the richest people in the system. After a global pandemic, in which frontline workers kept entire global economic order afloat, the rich are richer than ever, while workers are scrambling to pay the bills.

That’s why unions are popular. That’s why unions are happening.

There’s just no other recourse for such a wide-ranging, unfair, structurally entrenched bargain.

Developer experience: the fine art of making tools and platforms suck less

Love it or not, developer experience is buzz that’s here to stay. As the next generation of technology firms is born, while the last generation struggles to stay relevant, one of the best ways to grasp and hold a growth trajectory is simple: become a technological dependency of other firms.

To do this you have to make the case for integration.

This is harder than it sounds. Many technological abilities have become commodified. Developer experience is a differentiation play.

A successful developer experience strategy emphasizes tactics like:

  • Storytelling: Clear, ongoing narrative about the benefits of using a given technology so developers can understand where, why and how to apply it
  • Education: Approachable material—including documentation and sample code—for developing a mental model of how a technology works and how it integrates into developers’ systems and workflows
  • Onboarding: Low-friction, low-cost, high-speed onramps to test-drive a technology, allowing developers to get a feel for its abilities
  • Ergonomics: Optimization of workflows, tooling, standard libraries and API design to limit complexity, protect against error and quickly address the most common use cases

All of these areas can contribute to success, but none of them will matter without a strong commitment to a simple cause: providing accomplishment.

Developer experience is stewardship and conservation of precious cognitive resources, thereby maximizing accomplishment

You get success by creating success for other people. To do this, you have to maximize the yield of accomplishment for time, energy and attention invested in your technology.

Shitty technology that frustrates the developer is always at risk. The moment an alternative comes along that better respects their time and, especially, their sense of motivation, the incumbent technology has only inertia and lock-in to save it.

Example: Rust

Rust is a specialized language. It emphasizes performance, correctness and safety, so it can have a bit of a learning curve as a consequence of serving those goals.

But getting started with Rust is easy. A thoughtful system of trams exists to get you quickly to the foothills of its learning curve. Consider its get started page:

  • Automatic OS detection, providing a one-line terminal command to run an installation script
  • Quick briefings on the basic components of the Rust ecosystem, with links to editor-specific plugins so expert practitioners can easily use their existing tools
  • Introductions to common tasks, like creating new projects and adding dependencies
  • Sample code for a basic Hello, World! written in Rust

By covering all the necessary introductory topics, this intro to Rust:

  • Protects time and attention
  • Relieves new users of research burdens
  • Introduces tools and workflows
  • Creates a rough mental model of how to interact with the tooling
  • Generates a sense of accomplishment

It only takes a few minutes. Most of this time is the install script. By the end of a not-long page, you’ve gone from not having Rust at all to running your first program written with it.

Optimize your conversion funnel

At the top of the funnel are people who have problems and need to solve them.

At the bottom are people who have built solutions that depend on your technology.

Conversion funnels take effort, care and curiosity to optimize. You need to find the cheapest obstacle in the funnel that’s preventing your conversions, smooth it out, and repeat the process iteratively until your technology is easy—maybe even fun—to integrate.

A successful developer experience strategy is powerful because it creates irrefutable evidence of your technology’s value, in the form of people whose lives, projects and businesses are better.

For services and platforms whose billings scale with customer accomplishment, the math is obvious. Invest in your funnel, just like any other e-commerce business.

The invisible skill at the heart of every technologist

There’s a skill that’s core to success at all levels of software development, but it hides like dark matter: inferred, rather than observed, lurking beyond description and discussion. On a gut level, we know it’s necessary to the work, but we don’t often know how to teach it. You’ve either got it or you don’t in the perception of the average workplace.

Let’s call this skill investigative reasoning. Or, more simply: “working the problem,” as Apollo legend Gene Kranz might frame it.

(Let me know if MIT press has a $70 textbook explaining this, situating it in computing or engineering, and giving it a name.)

This pile of skills manifests as a reflex to make sense of an issue or failure, locate its causes, and design a fix. Without these abilities, you’ll be helpless in the stew of complex abstractions that define modern computing.

Consider a common software development workflow:

  • You write some code
  • It’s compiled, bundled, or otherwise transformed
  • The transformed code is transferred to a host device
  • The host runs the code

Failures can emerge at or after any of these steps. The code may have errors which prevent compilation. The code may have errors despite compiling fine, and they manifest only at runtime, during certain circumstances. There may be a failure in the systems that package and transfer the code, preventing it from running.

The experienced practitioner understands their job is to analyze the problem, develop a hypothesis for its source within the many layers of technology they’re using, and validate the hypothesis by checking logs, adding print statements, starting an interactive debugging session, or using other means of peering inside of the machine. Working the problem may require research: finding online discussions of similar issues, or consulting documentation.

But core to this activity for both the novice and the expert is a simple conviction: problems can be understood and investigation can yield solutions.

At any level, hobbyist to advanced professional, confidence in this premise is the indispensable requirement. Without the confidence to work a problem, it’s unlikely anyone can make progress in building and integrating technical systems.

Domain experience helps to work a problem. Every class of system, from microcontrollers to native applications to web apps, has its own set of quirks, its own unique abstractions and design decisions where trouble may sneak in. But working the problem is a skill that transcends domains. Fixing a broken software development toolchain can prepare you for troubleshooting the boiler in your basement.

The paranoia in hiring for software roles, I’m certain, is rooted in the fear that the organization will hire someone who is unable to work a problem, outsourcing their investigative reasoning tasks to others, depleting productivity per unit of headcount budget.

Working the problem is also a feature of seniority. Translating business requirements into software development tasks is an expression of this skill, as is understanding how technical tradeoffs (debt) in existing systems will impact the needs of new features. The more senior a practitioner is, the more likely they are to feel a sense of confidence working a problem no matter how many layers stack up—even when those layers are “non-technical,” derived from business or customer needs.

We laugh at the fixation on languages or frameworks in job postings, but even the most thoughtful organizations struggle to identify this set of activities as a job requirement, much less map their outcomes within the team’s problem space.

Not every person is the ideal practitioner to work a given problem. The startup costs in research, learning new tools, or picking up domain-specific expertise, may outstrip any return on investment. Still, I have to think that applying a more conscious growth mindset around this would give our industry better results in recruiting, professional development and retention than we have today.

How can we elevate our understanding of these skills? How can we instill the confidence needed to work a given problem? How can we better train people to be able investigators?

I think there’s a lot of missed opportunity here.

Developer experience: the basics

Periodically, tech Twitter swells with a discourse I find to be weirdly unproductive: people arguing about developer experience. Does it even exist? Does it matter?

I’m not going to recap the various head-scratchers in detail. But I do want to talk about what developer experience is, why companies invest in it, and what we all get for their trouble.

Spoilers: it's a problem of labor and software economics. A positive developer experience amplifies the return of effort invested building new software products.

The experience of getting things done

Put simply, developer experience is the sum of events that exist between identifying a requirement for a piece of software, and delivering code that satisfies it. Broadly, these events may be practical, emotional or social in nature.

Examples:

  • Referencing documentation to plan an integration of a third party service
  • Trying to install tools or libraries necessary to develop against a particular software framework
  • Getting frustrated because of an unfamiliar or undocumented design pattern
  • Successfully receiving help in a support forum when you get stuck

The practice of developer experience, of being deliberate in its design, is to identify the places of greatest leverage for clearing paths and relieving burdens. The objective is to improve adoption of a technology by making it easier to accomplish personal and business goals with it.

Developer tools: pickaxes in a gold rush

Selling developer tools is a straightforward business strategy. Rather than the risky bet of serving a broad consumer or cultural need, you can build technology that helps anyone who is building software be more successful.

Instead of trying to sell to hundreds of millions of users, you sell to a comparatively smaller handful of companies, typically by hooking their individual developers. If one of these customers succeeds, you succeed along with them, scaling your billings according to the growth of their business.

It sounds great, but there’s always a catch. In this case, your customers become a handful of technologists with a broad spectrum of experience levels and highly specialized needs. Your business success then rests upon a premise that’s easy to explain but harder to execute: making people more prosperous and effective because of your tools.

Leverage within the developer experience domain

How can you make people more successful and effective in accomplishing their goals? If we think of developer experience as the sum of all events between defining requirements and delivering them, we can identify some broadly recurring points of leverage.

Ergonomics and abstractions

Where the fingertips meet the keys, how does it feel to work with your tools? Does integration require painful, recurring boilerplate code, or can developers easily drop in your tools to solve a problem and keep moving to their unique implementation?

Is it easy to debug and inspect the state of your tools at runtime? When errors are thrown by your tools, is log output clear and descriptive, allowing further investigation and social troubleshooting on forums or Stack Overflow?

What is the everyday texture of life with your tool?

Tools that feel good to use obviously earn more loyalty, enthusiasm and word of mouth than tools that grate and frustrate.

Documentation, reference and education

How do people learn to use your tool? Do you provide clear documentation? Recipes? Tutorials?

What references exist for troubleshooting, debugging and discovery of features within your tool?

Thorough reference material makes it easier for developers to get the most out of everything you’ve built.

Community and ecosystem

Is there an active community experimenting and sharing their experiences with your tool? Is there a reliable, active, healthy venue where someone who is running into trouble can get help?

Is an eager community filling in gaps with their own tutorials, plugins, libraries and ergonomic improvements?

It’s easier to roll the dice on a new tool when you know that, should you need help, you’ll find a community that has your back.

Developer experience: the business cases

From these levers, we can identify the business cases for developer experience. In the context of the adopters of developer tools—whether individuals or teams—the question is whether a tool enhances their ability to be successful.

In a software production context, developer labor budget is among the costliest resources a business has to manage. Any tool that allows a business to get more for that budget is creating serious impact.

For purveyors of developer tools, the business case becomes clear as well. Success depends on adoption. You can improve adoption by addressing points of friction in existing developer workflows, and by making your tool’s experience more positive and productive than frustrating.

This doesn’t have to be hard

When we’re talking about developer experience, we’re talking about real things:

  • How people feel when using tools and making software
  • How effective they are in meeting their goals
  • How these factors converge into amplified productivity versus wasted effort
  • What leverage exists in your strategy to shift that balance more and more toward success for individual practitioners and businesses that might pay for your service

Developer experience is a process of shifting the economics of building software to be more favorable for every dollar or hour invested. There's a lot going on there, but conceptually, this doesn't have to be that hard.

Building a native macOS configurator for Adafruit's Macropad

Adafruit’s Macropad is an absolute hardware delight: a cluster of 12 keys, with individual lighting, plus an OLED screen, rotary encoder and even a speaker! While external, programmable keypads are common enough, there’s something to this combination of features that makes the Macropad more than the sum of its parts.

Top shot of powered on, assembled MacroPal keypad glowing rainbow colors.
Photo via Adafruit

It’s just a treat to hack on, and one of the rare technologies that makes me feel like a kid again, eager to experiment and try new things.

I want to walk you through a project I spun up recently, building a bit of macOS software to make my own Macropad easy to configure and tinker with:

I'll explain not just what I built, but how I approached solving the problems of integrating two very different systems speaking different languages. This stuff is essential to success in any sort of technology pursuit, but I always wish I saw more discussion of how we practioners do it.

Here’s my contribution.

Problem: configuration and iteration

I’ve been using programmable keyboards for years. Essential to success, for me, is iteration. Finding the most useful, comfortable layout of keys is a matter of trial and error, so I want to be able to experiment quickly with minimal friction.

So step one was finding whatever code existed to let me customize the Macropad’s key layouts.

Here I was in luck. Adafruit provides a terrific starting point:

MACROPAD Hotkeys

What’s brilliant about this example code is that fully utilizes the magic of the Macropad: turn its rotary encoder to move back and forth between pages of hotkeys. The page title and keymap labels are displayed on the OLED screen.

This seems simple and even obvious, but to me it’s a game changer because it makes an enormous breadth of keymaps viable. I don’t have to memorize key positions, so I can make pages upon pages of keys to solve whatever common problems I have.

There was just one wrinkle:

The configuration ergonomics.

Speaking to the Macropad

Macropad is built on an RP2040. This is a microcontroller: a tiny, specialized computer that you can code to solve a specific problem. Microcontrollers are amazing for hardware projects like the Macropad because they’re able to wrangle all kinds of electronic components—switches, lights, screens—and make them as easy to program as any software application on your computer.

Microcontrollers are hidden everywhere in your digital existence, but hobbyist platforms like RP2040, Arduino and others come with an ecosystem for conveniently programming and troubleshooting them without any specialized hardware. Plug them in via USB and you’re off to the races.

RP2040 supports code written two ways: C and Python. Adafruit’s example code used Python, so I followed their lead.

Most of this code was exactly what I wanted. Crucially, they'd solved all the problems of acting like a USB keyboard for me, picked out all the libraries I'd need. This is real research and experimentation effort saved.

But in terms of iterating on my keymaps, it didn’t quite hit the spot. Here’s how you define keymaps in their example project:

app = {                    # REQUIRED dict, must be named 'app'
    'name' : 'Mac Safari', # Application name
    'macros' : [           # List of button macros...
        # COLOR    LABEL    KEY SEQUENCE
        # 1st row ----------
        (0x004000, '< Back', [Keycode.COMMAND, '[']),
        (0x004000, 'Fwd >', [Keycode.COMMAND, ']']),
        (0x400000, 'Up', [Keycode.SHIFT, ' ']),      # Scroll up
        [...]

To tell the Macropad what to do for a given key, you provide an array of tuples, corresponding to each key, specifying:

  • hexadecimal value for backlight color
  • label string
  • an array of key presses

Thus,

(0x004000, '< Back', [Keycode.COMMAND, '['])

would create a key labeled < Back, with a green backlight, sending the key combination Command-[.

Look, this works, but there’s a lot of cognitive overhead involved. You have to build up these tuples, do RGB hexadecimal math for the colors, and make a tidy Python dictionary.

Better than a stick in the eye, but I’ve been spoiled by decades of GUI configurators for my input devices.

Still, this project had 90% of what I needed. If I could find a way to ingest configuration content that could replace this fiddly Python dictionary approach, I could build an interface for programming the keymaps however I wanted.

So that’s where I went next.

How do you say ‘red’?

After digging through this example code, I wrote down all of the things I needed to communicate to the Macropad to build custom layouts.

Some stuff was easy: if you want to specify a label that shows up on a screen, you use a string.

Other stuff needed a little checking. The colors were specified using hexadecimal. In other languages, I often see RGB hex handled as a string, but in this Python code, it took the form of 0xXXXXXX.

I’ve seen this around since I began using computers, but I never needed to actually know what it meant. In this context, the hex was being treated as an integer, expressed in hexadecimal notation. A little fiddling quickly confirmed the Macropad would treat it with the same behavior of hex colors I knew from the web, so that meant I didn’t need to handle colors in a unique way, which was a good start.

My next question was: can I convert a string into a hex integer in Python? StackOverflow, naturally, had me covered:

int("0xa", 16)

Initialize an int with a hex string, specify that its base is 16, and Python will do the rest.

So that’s the colors sorted. I could handle them like hex strings, externally. Again, strings are easy.

A thornier question remained: what about special keys, like option, command, etc? In the sample code they’re represented with some sort of Python variable, so I went on a hunt to find where these were defined.

Keycodes were represented like Keycode.COMMAND, and a comment in the configuration files offered this hint:

from adafruit_hid.keycode import Keycode # REQUIRED if using Keycode.* values

From that, I can infer:

  • Keycodes rely on an imported dependency
  • That dependency lives in a library called adafruit_hid

Sure enough, googling that led me to Adafruit’s GitHub project for a HID library. What’s HID? It stands for Human Interface Device, and it’s the USB standard that allows keyboards, mice and other peripherals to communicate on behalf of you and me.

From there it was a matter of digging around in the project structure until I found keycode.py, which contained a mapping of keycode names to the lower-level values the HID library relied on.

Spoiler: it was more hexadecimals:

    LEFT_CONTROL = 0xE0
    """Control modifier left of the spacebar"""
    CONTROL = LEFT_CONTROL
    """Alias for LEFT_CONTROL"""

Well, I’d just learned that hex values could travel easily as strings, so that’s not a big deal.

With these questions answered, I had the basics for translating values from outside the Macropad into input that it could properly interpret.

Next I’d need a medium to structure and carry these values.

Enter JSON

JSON is great: it’s become a universal standard, so every platform has a means of parsing it, including Python. It’s easy enough to inspect simple JSON with your own eyes, so that helps with troubleshooting and iterating.

I started by thinking through a structure for a JSON configuration payload:

Array: To contain pages of keymaps
	Dictionary: To define a page
		String: Page name
		Array: To contain keys
			Dictionary: To define a key
				String: Color (as hex)
				String: Label
				String: Macro

It would require a few iterations to get this to a final form, but this early structure was enough to get things moving.

Next I needed to learn enough Python to parse some JSON.

One of the most important habits I’ve picked up in a career of programming: isolate your experiments. Instead of trying to write and debug a JSON payload parser inside of the existing Hotkeys project, with the Macropad reporting errors, I used an isolated environment where I could get easy feedback about my bad code. Replit seems to provide the most robust, one-click, “let me just run some Python” experience online right now, so that’s where I ended up.

I badly wanted to figure out how to convert my JSON into proper Python objects, whose properties I could access with dot notation. This is how I’ve done JSON parsing in Swift for years, and it felt tidy and safe.

But after chasing my tail reading tutorials and Stack Overflow for awhile, I gave up and accessed everything using strings as dictionary keys. It was good enough. Ergonomics in the parsing code weren’t essential. Once it was up and working, I wouldn’t need to fiddle with it very much.

with open('macro.json') as f:
  pages = json.load(f)

for page in pages:

  keys = page['keys']

  imported_keys = []

  for key in keys:
    color_hex_string = "0x" + key['color']
    color_hex = int(color_hex_string, 16)
    macro = key['macro']
    [...]

It’s straightforward enough. The code loads a JSON file, iterates through the pages, pulls values out of the dictionaries it finds, builds key-specific tuples, doing conversions on hex code strings, and then plugs it all into the App class provided by the sample code, which represents a page. These are stored to an array.

Again, I don’t know any Python, so this took a bit of trial and error. Getting immediate feedback on syntax and errors from Replit’s environment helped me work through the bugs easily.

Once it all seemed to parse JSON properly, I dropped my code into the Hotkeys project, replacing the part that went looking for Python configuration values.

Macropad was happy, displaying a simple configuration I’d written in JSON by hand.

We were on our way.

A tool to create tools

With the basics of a working JSON structure, and a means for the Macropad to translate that JSON into keymaps, the table was set for real fun:

Building a user interface.

I’ve spent the last couple years writing a lot of SwiftUI code, and I was eager to try it out on macOS.

It’s surprising how straightforward it is to build multi-pane, hierarchical navigation in SwiftUI:

The old way, using AppKit and nib files, would have required loads more effort. But I had the basics of this up and working in a couple hours.

It’s not the most intuitive thing ever, but a quick google for swiftui macOS three column layout got me a terrific walkthrough that explained the process.

Next, I needed:

  • A data model to represent configurations, pages, keymaps, etc
  • A means of persisting those configurations so I could revise them later

For this, I turned to Core Data. It’s a polarizing technology. Many hate its GUI model editor, and it’s got plenty of complexity to manage. But as persistence strategies go, you can’t beat its out-of-the-box integration with SwiftUI. I’ve done enough time in the Core Data salt mines that I could quickly bang out the object graph I wanted and use it to generate JSON files. Best of all, the app could quietly store everything between sessions for easy iteration.

Modifiers were a challenge. In a perfect world, I could import keycode.py into Swift somehow and directly reference the values. My reading suggests this is possible, but I couldn’t sort out how. In the end, I used a spreadsheet to transform the Python code into Swift enum cases, making the hex values into strings:

enum AdafruitPythonHIDKeycode: String, CaseIterable, Identifiable {

    var id: String {
        return rawValue
    }
    
    case
    COMMAND = "0xE3",
    OPTION  = "0xE2",
    CONTROL  = "0xE0",
    SHIFT  = "0xE1",
    […]
}

As an enum, it was easy to iterate through all of the modifier keys and represent them in the UI. Storing them as strings made them easy to persist in Core Data. In theory I could convert them into plain integers, and send them around that way, but this seemed like less work to debug.

With navigation working and a data model specified, I went to work on the editor UI. My requirements were simple:

  • It had to visually represent the layout of the real thing
  • Editing had to be fast and easy

What I ended up with was a grid representing each key. Clicking a key let you edit its color (with a picker!), label text, and output. Thanks to the magic of SwiftUI, and a generous soul on StackOverflow, it was even easy to provide drag-and-drop reordering of the keys.

Clicking a button exported a json file, which could be saved directly to the Macropad—it shows up as a USB storage device on your computer.

Now I could bang out a new configuration in seconds, and what I saw in the editor was what I got on the Macropad.

“How’d you do that?”

In all, it was a weekend of effort to get this all rolling. At one point, showing it off, I was asked “How did you know how to do all of that?”

It’s a system integration problem! I’ve been doing that sort of thing for 20 years, so for a moment, I didn’t know where to start explaining. But to summarize the above, here’s how I approach this kind of challenge:

Find an existing, successful artifact

I need to start with something that already works. In this case, I found example code for the Macropad that was directionally aligned with my own goals. I’ve gotten so far in this game with example code. Code gives you pointers about how a system works, demonstrates its assumptions. It also gives you a starting point for your own approach.

No matter the system, in this age, you can usually find code that successfully interacts with it if you google hard enough.

Write down your unknowns

Once I’ve examined a working artifact, I usually end up with more questions than answers.

Hacking a project together is as much investigation as it is creation. I grab a pad of paper and keep track of my biggest unknowns to give that investigation its shape. After examining the example code, my biggest questions were:

  • How does it represent color?
  • How does it represent keycodes?
  • How do I parse JSON in Python?
  • How do I get feedback from the Macropad when things break?

If I found answers to these, I could build an external system that talked to the Macropad.

Learn how to communicate

The next step is asking “how does this system expect me to communicate my needs?”

Understanding the structure and format of data is essential. This step is equal parts research and experimentation. I need to find whatever documentation exists, even if it’s just source code, to understand the requirements and expectations of the system. Then, I need to try to create a working data structure of my own that the system will accept.

Build a bridge

Once you understand how to speak to a system, you need a means to reliably, repeatedly do so.

Here, I wrote a simple JSON parser to ingest and interpret external output into something Python could understand for one side of the bridge. On the other, I wrote Swift code that could generate JSON according to the expectations of that parser.

Fuck around, find out

With your unknowns revealed, a communication strategy understood, and a bridge constructed, you’re ready to start playing around. Experiment with different approaches until you get the results you’re looking for.

Sometimes you’ll break things. Use version control to keep track of your experiments, so you can always roll back to something you know is working.

Adafruit recommends the simple but effective Mu Python editor, which provides a serial connection to the Macropad. When I broke things, I could get hints about what went wrong that way through console log messages sent to the serial monitor.

Don't let perfect be the enemy of done

Throughout this process, there were "better" ways to accomplish what I wanted. I wish I could have parsed JSON into native Python classes, I wish I could have imported the keycode.py content more transparently into Swift. In a well-behaved Mac app, you can double-click a list item to edit its name. Still not sure how to do it in SwiftUI.

It's easy to get bogged down in the perfect solution, but I try to prioritize progress over perfection. Do the ugly, hacky thing first, then keep moving. If it comes to bite you later, you can rewrite it. Probably you'll have a lot more context later, so the deeper solution will be better informed than if you'd tried it from jump.

What about the configurator?

Sure, here’s the code:

macOS Project

Macropad code, as adapted from the MACROPAD Hotkeys project

Stuff I might add in the future:

  • Properly signed binary builds of the configurator (right now you'll need Xcode)
  • JSON import
  • Multi-part macros with delays
  • Undo support
  • Direct user input capture of keystrokes

Maybe this will inspire you to write a web application that does the same things.


Hope this look inside the hack was interesting. Drop me a line with any questions you’ve got about how tech gets built. Much love to Adafruit, which makes the coolest electronics hobby stuff around. Projects like this Macropad remind me why I fell in love with technology:

  • Exploration
  • Experimentation
  • Magic

I have the life and career I have because once upon a time, I learned how much fun it was to make things that talked to each other.