Skip to main content Redeem Tomorrow
I used to be excited about the future. Let's bring that back.
About Get my help

Retrospective on a dying technology cycle, part 2: Open Source and the Cloud

[Previously: Mobile]

Though it rescued me from insolvency, I grew to hate my job at Aurora Feint. Two months into the gig, Apple Sherlocked us, sending the company into a series of endless pivots that lasted four to six weeks each.

The engineers were crunching, working evenings and weekends, only to throw away their work and start again, seemingly based on how well one press release or another gained attention. Meanwhile, I was far from the action in an ill-defined product management role. While I knew the emerging mobile product space as well as anyone, I had limited vision into the various rituals of Silicon Valley.

What I really wanted was to build mobile software.

I interviewed for months, but with no pedigree and a strange path into technology through self-teaching, I didn’t get very far.

After a particularly dispiriting day at the office, I sniped the attention of Adam Goldstein and Steve Huffman, then founders of a YC-funded travel search startup called Hipmunk. I wrote a flattering blog post about their product, posted it to Hacker News, and within weeks had a new job: leading mobile product design and engineering for a seed-stage startup.

They never asked for a resume. They checked out my existing work on the App Store and made me an offer.

I was the third employee. The entire company of five could fit into a San Francisco apartment’s living room. And did, regularly, in its first few months.

Despite the small crew and modest quarters, thousands of people were buying airfare through Hipmunk already. Its great innovation was visualizing your flight options on a colorful Gantt chart, ranked by “Agony”—factors like layovers and overall flight time.

Two things gave this team enough leverage to do it: open source software and cloud computing. A credit card was all they needed to provision server capacity. An internet connection gave them access to all the software they needed to make the servers productive, laying the foundation for building Hipmunk’s unique approach to finding flights.

What is Open Source, anyway?

Open source has permanently changed the nature of human productivity.

Open source describes code you can use, according to some basic licensing terms, for free. Vast communities may contribute to open source projects, or they may be labors of love by one or a handful of coders working in their spare time.

In practice, open source code is encapsulated labor.

One of the fundamental economic levers of software in a globally-connected community is that it scales at zero marginal cost. Once the software exists, it’s nearly free duplicate it.

Thus, if you have software that solves a common problem—provides an operating system, handles web requests, manages data, runs code—you can generalize it, and then anyone can use it to solve that category of problem in their own project. This combination of open source projects, called a “stack,” is sometimes packaged together.

Such open source stacks made the last cycle possible. The Facebook empire could be founded in a dorm room because Zuckerberg had access to Linux (OS), Apache (web server), MySQL (database), and php (language/runtime) as a unified package that solved the problem of getting you productive on the web.

This represented tens of thousands of hours of labor that startups don’t have to recruit or pay for, or even wait for, dramatically accelerating their time to market.

The details of serving HTTP requests or handling database queries aren’t central to a startup’s business value—unless you’re making a platform play where your performance on these categories of tasks can be sold back to other firms. But for most, there’s no advantage to reinventing those wheels, they just have to work well. Instead, borrowing known-good implementations can quickly provide the foundations needed to build differentiated experiences.

In Facebook’s case, this was a minimalist social platform, for Hipmunk this was visual travel search, while Uber dispatched cars. There was no innovation to be found merely existing on the web. Instead, to survive you had to be on the web while also providing unique, differentiated value.

Open source is an informal commons of these known-good implementations, regularly refined and improved. Any party finding a project lacking can lobby for refinement, including proposing direct changes to the code itself.

I would argue there is societal impact from this that rivals other epochal moments in information technology, like the advent of writing or the discovery of algorithms. Being able to store, replicate and share generalized technical labor is a permanent reorganization of how millions of people work. While capitalists were among the first to seize on its power, over decades and centuries it may permeate much more than business.

The power of open source was constantly animating the last cycle. Node.js, a server-side runtime for JavaScript first released in 2009, took one of the most well-known languages in the world and made it viable for building both the user interface components of a web application and its backend server as well. Thus open source could not just encapsulate labor, but amplify it in motion as well. Suddenly, many more people knew a language that could be used to build complex web services. As a result, Node became among the most popular strategies hobbyists and startups alike.

Meanwhile, it wasn’t enough for code to be easy to build on. The code actually has to live somewhere. While open source was a crucial engine of the last cycle, cloud computing running this open source code was just as essential.

It’s in the cloud

Information is physical.

It needs somewhere to live. It needs to be transformed, using electricity and silicon. It needs to be transported, using pulses of light on glass fibers.

Software requires infrastructure.

Historically, this had meant shocking amounts of capital expenditures—thousands of dollars for server hardware—along with ongoing operating expenditures, as you leased space in larger facilities with great connectivity to host these servers.

Scaling was painful. More capacity meant buying more hardware, and installing it, which took time. But the internet is fast, and demand can spike much more quickly than you could provision these things.

Amazon, craving massive scale, had to solve all these problems at unique cost. They needed enough slack in their systems to maintain performance at peak moments, like holiday sales, back-to-school, and other seasonal surges. With the size of Amazon’s engineering division, meanwhile, this capacity needed to be factored into tidy, composable services with clear interfaces and rigorous documentation, allowing any team to easily integrate Amazon’s computing fleet into their plans and projects.

Capital abhors an under-leveraged asset, so they decided to sell metered access to these resources. Though Amazon was the first to use this approach, today they compete with Microsoft, Google, IBM and others for corporate cloud budgets all over the world.

With access to “the cloud” just a credit card away, it’s easy for small teams to get started on a web-based project. While scaling is commercially much easier than ever—you just pay more to your cloud vendor, then get more capacity—it can still be a complex technical lift.

The arcane art of distributing computing tasks between individual machines, while still producing correct, coherent output across large numbers of users, requires specialist labor and insight. The cloud isn’t magic. Teams have to be thoughtful to make the most of its power, and without careful planning, costs can grow devastating.

Still, this power is significant. The cloud can itself be programmed, with code that makes scaling decisions on the fly. Code can decide to increase or trim the pool of computing capacity without human intervention, adjusting the resources available to create maximum customer impact and system resiliency. Netflix is a notable specimen on this point, as one of the earliest large-scale adopters of Amazon’s cloud services. They created an infrastructure fabric that is designed to tolerate failure and self-heal. They even built a technology called “Chaos Monkey” to continually damage these systems. This forces their engineers to build with resilience in mind, knowing that at any moment, some infrastructure dependency might go offline.

(Chaos Monkey is, naturally, available as open source.)

For companies in infancy, like Hipmunk, the cloud opened the path to a fast-iterating business. For companies entering maturity, like Netflix, the cloud provided a platform for stability, resilience, global presence, and scale.

The world had never seen anything like it. In fact, there was early skepticism about this approach, before it came to dominate.

Today, the cloud is a fact of life. Few would even bother starting a new project without it. What was once transformational is now priced into our expectations of reality.

Next

Hipmunk would not become a galloping success. It sold to Concur in 2016, eventually scrapped for parts. Steve Huffman would return to lead Reddit, his first startup. Reddit sold to Condé Nast for peanuts, only to spin back out as an independent venture as, stubbornly, it just wouldn’t die.

Meanwhile, six months after my departure, the renamed OpenFeint was acquired for $100m by a Japanese mobile company in need of next-generation mobile technology. I didn’t get a dime of the action—I left before my cliff kicked in, and I couldn’t have afforded to exercise the options even if I they were open to me.

That’s the circle of life in venture capital. Not every bet is a winner, death is common, and sometimes acquisition is bitter.

But when the winners win, they can win big. After a couple of years, I left Hipmunk for a couple more gigs leading mobile teams. Under all the VC pressure and SF’s punishing cost of living, I burned out hard.

Somehow, barely able to get out of bed most mornings, I landed on a unicorn. In the next piece, we’ll explore how venture capital was the fuel of the last cycle, creating growth in a zero interest rate global economy by harnessing startups that, impossibly, created billions of dollars in value.


Retrospective on a dying technology cycle, part 2: Open Source and the Cloud Retrospective on a dying technology cycle, part 2: Open Source and the Cloud