Untangling Dependencies at Amazon

Image not Found

As companies scale, they slow down. Teams spend more time coordinating and less time building. Every new feature overlaps and intertwine with features in other teams’ roadmaps. Teams need more meetings, people, and time to complete their work. The organization get tangled in a network of dependencies. However, untangling dependencies is one of the most challenging problems in software development.

This path is not inevitable. The most critical element determining team success is autonomy—the team’s ability to decide how to best fulfill its mission and deliver value independently.

Autonomy is not pre-ordained. It’s a leadership choice

Teams can’t wake up one morning and start moving fast. They require mechanisms and a culture that enables them to be autonomous. Over the last two decades, Amazon has organized itself to reduce technical and organizational dependencies. In this article, I will list three principles Amazon uses to keep teams independent: team autonomy, decentralized decision-making, and architecture principles.

Team autonomy: Amazon restructured its organizations into small, autonomous teams, small enough that they can be fed with two pizzas. A network of small, autonomous teams helps Amazon to distribute invention throughout the company. To achieve that, teams must be able to own their problem space end-to-end.

Team autonomy
Two-pizza teamsOrganize in small, empowered teams
Single-threaded modelFocus singularly in just one initiative end-to-end
You build it, you run itOwn the entire lifecycle of the product
Team mitosis modelPrevent two-pizza teams from becoming thee-pizza teams
Away team modelDecouple roadmap dependencies across teams
Tolerate, then Eliminate DuplicationPrefer two solutions than none, one is better than two.
Communication is terribleReduce overhead of external communication

We disproportionally index in hiring builders, and then we organize them in small separable and autonomous teams as we possibly can and let them own their own destiny

Andy Jassy, CEO of Amazon

Decentralized decision-making: Amazon pushed down most decisions to frontline teams to decentralize innovation. High-velocity decision-making is a critical aspect of nimble companies. In some enterprises, command-and-control is the way that work gets done. Ensuring that essential decisions only get made by those at the top maintains high-quality decisions. However, they make high-quality decisions slowly.

Decentralized decision-making
Increasing Spans, Decreasing LayersReduce the distance between decision-makers and customers
One-way and two-way door decisionsMake reversible decision fast, dive deep in consequential decisions
TenetsAct as the North star up to codify decision-making
Bar raisersPrevents the urgency to co-opt long-term thinking
Rapid EscalationUse escalations to accelerate alignment
ExperimentationAllows teams to experiment early and frequently
The Institutional yesPrevent vetocracy from switching off independent ideas

[Decentralized] distribution of invention throughout the company — not limited to the company’s senior leaders — is the only way to get robust, high-throughput innovation.

Jeff Bezos, Amazon founder - Annual shareholder letter 2013

Architecture principles: Amazon rearchitected its applications into small pieces called “services”. The distributed computed manifesto established the principles to help teams work independently. Amazon Simple Storage Service launched with ten design tenets that acted as a North Star for service teams to grow their system from eight to more than two hundred fifty distributed services over the last fifteen years.

Architecture principles
Architecture TenetsAct as a North star from which to evaluate decisions
Service Oriented ArchitectureRestructure applications into smaller pieces called “services”
API MandateCommunicate only through interfaces over the network
Self-service platformsPrevent well-meaning gatekeepers from slowing down innovation
Automated release pipelinesAutomate safe, hands-off deployments

We created a blueprint for change with our “Distributed Computing Manifesto.” This was an internal document describing a new architecture. With this manifesto, we began restructuring our application into smaller pieces called “services” that enabled us to scale Amazon dramatically.

Werner Vogels, CTO of Amazon

To operate efficiently at scale, organizations need to embrace mechanisms that best fit their position

Amazon uses these principles in the context of its Day 1 culture. Other organizations have similar mechanisms tailored to the culture and ways of working.

TwillioMove fast duplicate workTeam Autonomy
AppleDirectly Responsible IndividualsDecentralized decision-making
GitlabEscalate to unblockDecentralized decision-making
CoinbaseAPIs instead of meetingsArchitecture principles

Larger companies also get slowed down by endless meetings around prioritization and feature requests. We need to move to a model where all product and engineering teams (not just shared services) publish APIs so that other teams can benefit from what they’re building without ever needing to schedule a meeting. In other words, they need to productize their services and allow other teams to use them in a self-service way.

Brian Armstrong, CEO of Coinbase

Untangling dependencies require end-to-end ownership

Every dependency creates drag, increasing the chances of being delayed. At Amazon, leaders own driving the process to remove dependencies. They need to identify and relentlessly address every potentially impacting dependency—a failure due to a dependency is a leadership failure. In 2003, during an S-Team meeting, Jeff Bezos broke down the process of managing dependencies into three steps. Every team should:

  1. Take over their dependencies, so they don’t rely on others
  2. Negotiate unambiguous commitments from others, If owning the dependency is impossible
  3. Create hedges wherever possible. Every dependency needs to have a fallback plan

The most successful teams invested much of their early time in removing dependencies and building “instrumentation”—our term for infrastructure used to measure every critical action—before they began to innovate, meaning, add new features.

Bill Car, Vice President of Digital Media at Amazon

In conclusion

Growth creates complexity, and complexity is the silent killer of speed. Organizations must find ways to retain velocity through the appropriate mechanisms to avoid becoming complacent and irrelevant over time. Companies can trade coordination and alignment for autonomy. However, autonomy has trade-offs (e.g., duplication of work between teams, re-learn).

To encourage speed and innovation, organizations need to build a network of autonomous teams, a mindset of ownership end-to-end, a culture that encourages distributed decision-making for more than just the senior leaders, and fit theri technical architecture around autonomy.

Breaking down our organization and application structures was a bold idea, but it worked. We were able to innovate for our customers at a much faster rate, and we’ve gone from deploying dozens of feature deployments each year to millions, as Amazon has grown.

Werner Vogels, CTO of Amazon Web Services

I will dive deeper into each mechanism I listed above in future posts. In the meantime, reach out via LinkedIn, Mastodon or Twitter

comments powered by Disqus

You May Also Like