Loading...

How large successful companies achieve agility at scale

This is an account of how large successful companies like Microsoft, Amazon, Google, Facebook, Spotify, LinkedIn, Netflix and HP LaserJet Firmware achieved agility at scale with tangible and lasting business benefits.
It includes a summary of the lessons to be learned.

Feels free to copy and distribute this post as you please. Get in touch to suggest any improvements.

Compare this account with The results of companies migrating to heavy-weight scaled frameworks and off-the-shelf platforms.

Similarities, and lessons to be learned

Microsoft, Amazon, Google, Facebook, Spotify, LinkedIn, Netflix and HP LaserJet Firmware: all these big and successful companies work to deliver digital services and/or digital products, or physical products with embedded software.

This paragraph is a summary of the similarities and the differences among how these successful companies achieved Agility at scale. They are intended as lessons that can inspire other companies also trying to achieve Agility at scale.

In the following paragraphs you find a detailed account for each of the companies including the references to the main sources used.

Click on the summary image below to zoom (WoW acronym stands for Ways Of Working).

Comparison of top companies scaling approaches

These successful companies have achieved Agility at scale, with tangible and lasting benefits for their business.
Agility here is intended as being fast, dynamic, innovative, flexible, adaptive, and resilient.
It means they all achieved agility at scale in their business, in their organisation and in their technology.
At Scale, because they employ from a few hundred to several thousand software developers and engineers;  and some of them develop large software systems that require many of those developers and engineers to work together on the same piece of software/delivery initiative.
With tangible and lasting benefits for their business, as is proven by their undisputable successes.


Some of those companies had that agility from the outset and maintained it while they grow (Google, Facebook, Spotify, Amazon) while others at some point in time went through some kind of change or transformation to achieve or recover such degree of agility (Microsoft, LinkedIn, Netflix and HP LaserJet Firmware).

Their pursuit of agility has been driven both bottom-up and from the top, with a lot of autonomy as well as responsibility for the teams. The balance between the bottom-up and the top drive vary from company to company.  People, their autonomy, their mastery, their involvement, and team-work have been fundamental ingredients.

None of these companies even remotely refers to, uses practices from, or takes inspiration from any one of the commercial scaled agile frameworks (e.g. SAFe, DaD, or even LeSS).
None of these companies adhere to just one of the official Agile framework, not by the book, not by imitation, nor in a prescriptive or uniform and standard way for all the teams with a cookie-cutter or cut-and-paste approach.

All these companies have grown their own ways of working internally, in a continuous process of experimentation-learning-adaptation-evolution, in order to suit their specific circumstances and needs (no scaling by imitation, no copycats).
All these companies’ ways of woking take inspiration or are congruent with many of the Lean and Agile principles, values and mindset. With a few exceptions for two of them on sustainable pace, psychological safety, and daily collaboration with the business.
In their pursuit of agility, all these companies have a strong focus on technical excellence with a strong engineering culture. All of them consider the company culture as an important ingredient of their agility (suggested reading: Culture and cultural change frameworks and ideas).
All these companies’ have some technical practices at scale that have their roots in Agile and specifically in Extreme Programming (e.g. Trunk Based Development, Continuous Integration, Test Automation, Continuous Delivery).

They are still continuously evolving their own ways of working (no transformation programs, no target operating models or any end-state or end-date, no distinction between improving and delivering, just every-day continuous improvement).

Some of these companies (Microsoft, Spotify, HP LaserJet Firmware) explicitly define themselves as Agile and explicitly adopt ideas and practices from several Agile frameworks (e.g. Extreme Programming, Scrum, Kanban, Lean Startup). The other companies do not explicitly define themselves Agile and do not have any explicit company-wide policy about Agile, while Agile exists in pockets in their organisation because of the autonomy of the teams.
None of these companies talks about scaling Agile, some talk about Agile at scale. 

In conclusion, for all these companies, their agility has been instrumental for their business, financial and commercial success, and for keeping their edge and innovativeness while growing and at scale.

These stories shared below provide good learnings about things that worked very well for these companies. Correlation does not imply causation, so do not just copy these as recipes, instead use these learnings to inform and inspire the experimentation in your company with an experiment-learn-adapt-evolve approach.

Microsoft

The Agile journey at Microsoft has been underway for some ten years starting from 2008 and continues to evolve.

Over these ten years, the journey has been:

  • Bottom-up for the first 7 years, at the team level: began with one team, then three teams, then 25 teams, then many teams.  One of the divisions, Microsoft’s Visual Studio group, had about 4,000 developers.
  • Then Top got involved: the new CEO embraced it, and it spread across the entire firm and steadily became part of the culture.

The ways of working have grown internally without following or taking inspiration from any of the commercial scaled frameworks. The approach indeed has been to be Agile at scale, not to scale Agile.
Initially, there was a focus on Agile technical practices (Continuous Integration, Continuous Delivery, keeping on top of technical debt), and Scrum together with an increased focus on customers.  After that the emphasis moved on an Agile mindset, a culture of trust and business results, keeping the customers involved, and giving the team the right balance of autonomy and alignment. 

Image from Steve Denning
Business Agility in the Y axes is intended as Agility for the whole organisation.

Microsoft’s rediscovered agility has been instrumental in its turnaround. Microsoft overcame the disease of bureaucracy, took quick and bold decisions: they de-prioritized Windows, bet big on the “cloud”, opened up to and embraced perceived rivals such as open-source, Linux and open standard making Visual Studio cool among developers and successful.

Below the suggested content.

Big Picture Article: How Microsoft Vanquished Bureaucracy With Agile 
– https://www.forbes.com/sites/stevedenning/2019/08/23/how-microsoft-vanquished-bureaucracy-with-agile/

Article: Microsoft’s 16 Keys To Being Agile At Scale
– https://www.forbes.com/sites/stevedenning/2015/10/29/microsofts-sixteen-keys-to-becoming-agile-at-scale/

Video: Account from Aaron Bjork of the initial journey at the team level
– https://azure.microsoft.com/en-gb/resources/videos/agile-at-microsoft/

Videos and articles: DevOps at Microsoft
– http://stories.visualstudio.com/devops/

Amazon

Unlike the other companies mentioned here, the adoption of the modern post-bureaucratic ways of working in Amazon has been led from the top. Amazon in some ways has adopted such modern ways of working from the outset as a public company since 1997, and continuously evolves them. Some of the elements of Amazon’s approach are:

  • an obsession with adding value to customers, 
  • quick decision-making,
  • a startup culture that value creativity, continuous innovation, and growth, and it is risk-taking and learns from failures.

It was 2002 when Jeff Bezos introduced and quickly implemented the frequently quoted “two-pizza teams”: small entrepreneurial teams aligned to the strategic priorities of the organisation, that almost independently build, run, support and ultimately own a set of services/product capabilities.

The transition to “two-pizza teams” included a fundamental technical change that enabled teams to quickly innovate at scale with a strong customer focus: Amazon made a huge technical effort to move the technical platform, infrastructure and architecture from a two-tier monolith to a fully-distributed, decentralised, services platform. Some identify this pivotal moment as when Amazon achieved its Agility, but many argue that the seeds of agility were there from the beginning and this adaptation was an expression of that.

Image from Steve Denning
Business Agility in the Y axes is intended as Agility for the whole organisation.

Amazon modern and continuously evolving ways of working are grown internally. 

Amazon doesn’t call itself “Agile” though its approach enacts many lean and agile ideas, principles and approaches for example around the team, customers, products, prioritisation, and technical agility (see for example Scrum, Extreme Programming, Lean startup) with a couple of peculiarities:

  • In Amazon, the continuous evolution of the company’s ways of working has been led from the top (autonomy and ownership of the outcomes is at the teams level),
  • Amazon is known as a very demanding place to work. It’s not for everyone, you could say the company is ‘aggressive.’ 

The latter point is not aligned with the Agile principle of sustainable pace of work, nor with psychological safety that is one of the ingredients to support individuals as per one of the agile principles.

In the Amazon approach, there are no implicit or explicit references or influences to and from any commercial scaled agile framework. 

Amazon’s ways of working, culture and ultimately agility have been the foundations of its commercial and financial successes in very disparate markets and products, e.g. AWS and Echo and Kindle.

Below the suggested content.

Article: How Amazon Became Agile
– https://www.forbes.com/sites/stevedenning/2019/06/02/how-amazon-became-agile/

Pdf: A Conversation with Werner Vogels (Amazon CTO)
https://dl.acm.org/doi/pdf/10.1145/1142055.1142065?download=true

Article: How Mapping The Agile Transformation Journey Points The Way To Continuous Innovation (paragraph The Amazon Story)
– https://www.forbes.com/sites/stevedenning/2019/03/17/how-mapping-the-agile-transformation-journey-points-the-way-to-continuous-innovation/#60edc1ff4b24

Post: Amazon Leadership Principles
– https://www.amazon.jobs/en/principles

Article: How Amazon Uses Agile Team Structures and Adaptive Practices to Innovate on Behalf of Customers 
https://www.hrps.org/resources/people-strategy-journal/spring2019/pages/galetti-golden.aspx

Google

Google is a tech company with more than 40 offices around the world.

People at Google people don’t say “IT” (information technology) because the business & tech work closely together, and IT is an integral part of the business.

Google does large-scale software development: as reported in posts from 2010-2013, they have 15.000+ developers working on a codebase of more than 100 million lines of code.
Because of the scale, Google needs to develop its own tools and infrastructure.

In more technical terms Google practice Trunk Based Development: it has a single monolithic code tree, with all the submission done on the head and all the builds and the releases done from source. Google also practice Continuous Integration at scale, has extensive test automation with 100+ million test cases run per day, and more than 50000 builds per day on average.

Technical practices used by Google such as Trunk Based Development, Continuous Integration, and Test Automation, have their roots in Agile software development and more specifically in Extreme Programming. But Google does not make any explicit references to Agile in relation to their ways of working while taking some inspiration or using the evolutions of Extreme Programming.

In relation to scaling their ways of working instead there is no reference at all, no inspiration, no use of any idea or evolutions from any of the commercial agile scaled frameworks.

There is a lot of freedom and organic emergence of tools, practices and processes from those doing the work, in Google.
There are only a few processes and practices and tools mandated in Google, while engineers have a lot of autonomy.
At an individual delivery initiative level (the original paper uses the word project), uniformity is rarely mandated and adoption of tools and processes is left to an internal “market” to decide. A large portion of Google tools is developed by motivated individuals to solve local challenges. Similarly, the process is tailored specifically to the initiative. While this leads to a healthy amount of chaos, good ideas tend to spread quickly, because they have been proven useful by others. Engineers decide what’s best for engineering.

Agility has been fundamental for Google to remain innovative and fast while growing at an unprecedented scale, and ultimately maintaining its technical supremacy that started with taming the sea of information, continued with reinventing emails, mapping the world, … , and now leading the progress of Artificial Intelligence applications.

Below the suggested content.

Article: The five keys to a successful Google team (while doubts have been raised on the validity of this study, its account still gives a snapshot of teams at Google)
– https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/

Article: The Design Sprint (see the final paragraph “A Brief History of the Design Sprint”)
– https://www.thesprintbook.com/how

Pdf: Continuous Integration at Google Scale
– https://eclipsecon.org/2013/sites/eclipsecon.org.2013/files/2013-03-24%20Continuous%20Integration%20at%20Google%20Scale.pdf

Article: Google’s Scaled Trunk-Based Development
https://paulhammant.com/2013/05/06/googles-scaled-trunk-based-development/

Article: The Birth of Automated Testing at Google in 2005
– https://dzone.com/articles/case-study-the-birth-of-automated-testing-at-googl

Article: Test Mercenaries (account of the 2006-2009 Google initiative to improve test automation)
– https://mike-bland.com/2012/07/10/test-mercenaries.html

Online Book: Google Site Reliability Engineering
– https://landing.google.com/sre/sre-book/toc/

Pdf: Google’s Innovation Factory: Testing, Culture, And Infrastructure
– https://research.google/pubs/pub41672/

Article: Is Google Going Agile?
– https://adtmag.com/articles/2010/07/30/is-google-going-agile.aspx

Facebook

Facebook in 2019 had 43,000 employees and 2.45 billion monthly active users (Source: Facebook 10/30/19). Since its launch in 2004, Facebook is continuously growing, and so their ways of working are continuously changing and evolving.

Facebook is not only a large company, but it is also innovative, fast, flexible and resilient. It demonstrates agility at scale.

The company culture is very engineering-driven. Engineers contribute to defining the direction of the product, or product’s capabilities, they work at, and the related roadmap. Engineers also have degrees of autonomy in their ways of working and the evolution of the ways of working, that is informal and organic.

Facebook to deal with the speed of growth and its scale, has to develop some of its tools. For example, Facebook employs a  single company-wide source code repository, a mono repo, that handles thousands of commits a week across hundreds of thousands of files from a codebase of several million lines of code.

In terms of technical practices, Facebook practices Continuous Integration at scale, Test Automation, Trunk Based Development, and Continuous Delivery releasing into production many times per day. Facebook also practices collective code-ownership: any engineer can modify any part of FB’s code base and check-in at-will.
All these technical practices have their roots in Agile Software Development and more specifically in Extreme Programming.

In Facebook, the work is organised in small cross-functional teams that own and are responsible for a product or product capabilities. This approach has been popularised by Agile Software Development. Some teams have been reported to adopt some of the practices from some Agile frameworks (e.g. daily standup).
Facebook does not make any explicit references to Agile in relation to their ways of working, in any of their communications, while taking some inspirations or using the evolutions of Extreme Programming practices.

In relation to scaling their ways of working instead there is no reference at all, no inspiration, no use of any idea or evolutions from any of the commercial agile scaled frameworks.

Below the suggested content.

Product engineering at Facebook (notes on cross-functional teams)
https://engineering.fb.com/android/product-engineering-at-facebook/

See paragraphs ‘Small, Independent Teams’, and ‘Control and Responsibility’
https://www.facebook.com/note.php?note_id=409881258919

Facebook’s Trunk-Based Development
https://paulhammant.com/2013/03/13/facebook-tbd-take-2/

Scaling Mercurial at Facebook
https://engineering.fb.com/core-data/scaling-mercurial-at-facebook/

Ship early and ship twice as often
https://engineering.fb.com/uncategorized/ship-early-and-ship-twice-as-often/

See paragraph ‘Continuous Integration testing’
https://trunkbaseddevelopment.com/vcs-features/

Facebook engineering
https://engineering.fb.com/

Spotify

Spotify is another well-known example of a large tech company adopting Agile ways of working. They did it while growing very quickly. Spotify was founded in 2006. In 2012 they had hundreds of developers divided into 30 agile teams spread over 4 cities in 3 timezones. In 2014 they had 1500 employees of which 700 in Tech, Product and Design. In 2016,  2000+ employees.

To be Agile at scale they did not use any commercial scaled agile frameworks.

They grow their ways of working internally. In this ongoing journey, they have been inspired by Lean and Agile principles among other things. Teams used Scrum, Kanban, and Lean Startup. And teams had a strong focus on the engineering aspects. Some of the Agile technical practices (such as TDD, Pair programming and Simple design) were used by some teams but not from other, other Agile technical practices (like Continuous Integration, Continuous Delivery and DevOps for example) were more widespread.

Spotify was and is not an Agile nirvana. For example in the squads (the teams) there was an imbalance between the decision right of the Product Owner and the technical counterpart leading to technical debt for some teams. The allocation of some line management responsibilities outside the teams lead to a matrix management structure with the well-known downsides. Some teams become too specialised (as opposed to owning end-to-end a product area or function) leading to many cross-team dependencies when dealing with bigger features. Agile Coaches were stretched too thin not giving enough support to teams to learn how to align and how to collaborate in the face of cross-teams dependencies.

Between 2012 and 2014 Spotify shared a  simplified description of their ways of working and the several recurring “patterns” they have observed there. Those descriptions were a snapshot at that specific time of a quickly and always evolving company. They have never been intended to be used as a model to be replicated internally for every corner of the organisation (no cookie-cutters) or to be imitated or copied externally by other companies (no copy-and-paste). Because that is fundamentally in contrast with how agile ways of working grow and evolve. 

Below the suggested content.

Pdf: Scaling Agile at Spotify
https://blog.crisp.se/2012/11/14/henrikkniberg/scaling-agile-at-spotify

Video: Spotify Engineering culture
https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/
https://labs.spotify.com/2014/09/20/spotify-engineering-culture-part-2/

Challenges at Spotify
https://www.jeremiahlee.com/posts/failed-squad-goals/

LinkedIn

LinkedIn was founded in 2002 and launched in 2003. As reported in 2016, the company has around 9,200 employees with 33 global offices.

Starting from 2011 LinkedIn switched from a waterfall-like development process to modern ways of working, going from a system that requires a month to release new features to one that pushes updates into production multiple times per day. Kevin Scott, the senior vice president of engineering, and his team of programmers had a key role in this switch.

LinkedIn’s big switch has been linked to very concrete and visible financial success, and business capabilities.

LinkedIn’s new and modern ways of working revolve around Continuous Delivery, with Test Automation, and Trunk Based Development. All these practices have their roots in Agile software development and more specifically in the Extreme Programming Agile framework. LinkedIn indeed takes inspiration from Agile frameworks, such as Extreme Programming (Scrum and Kanban are also mentioned in some of their job postings), while it does not formally adopt a specific Agile framework.

Furthermore in LinkedIn’s ways of working there is no reference at all, no inspiration, no use of any idea or evolutions from any of the commercial agile scaled frameworks.

In LinkedIn, there is a strong Engineering culture and a culture of tech excellence. With teams having degrees of autonomy on their ways of working.

LinkedIn’s new ways of working required a huge commitment from both the business and the tech, at some point the process required halting all development for two months as LinkedIn trained staff, migrated old code, and built out their own automated tools and infrastructure needed to make the new system work.

The benefits for the business have been immediately visible.
Examples of features that have been quickly created and released thanks to LinkedIn’s new and modern ways of working, include revamped company pages, revamped profile pages, overhauled notifications, a redesigned homepage, comments and likes on news pages, “people you should hire” suggestion box for recruiters, mobile apps, a job listings app, blogging, … and the list goes on. Other feature would have been simply impossible to create without LinkedIn transforming his ways of working, examples include endorsements, the new influencers product, the new version of profile and the several things in mobile.


Below the suggested content.

Article: The Software Revolution Behind LinkedIn’s Gushing Profits
https://www.wired.com/2013/04/linkedin-software-revolution/

Post: Engineering Culture at LinkedIn
https://engineering.linkedin.com/engineering-culture/engineering-culture-linkedin

Post: Scaling Decision-Making Across Teams within LinkedIn Engineering
https://engineering.linkedin.com/blog/2018/03/scaling-decision-making-across-teams-within-linkedin-engineering

Posts: LinkedIn Engineering blog etc
https://engineering.linkedin.com/

Post: How Agile Helps the GTS Team Get Laser Focus
https://engineering.linkedin.com/blog/2015/11/how-agile-helps-the-gts-team-get-laser-focus

Netflix

Netflix was launched in 1998 with a focus on the initial DVD rental business. It is in 2010 when Netflix expanded its business with the introduction of streaming media, and internationally. In 2012 entered the content-production industry.
Since after the dot-com bubble burst in 2000, Netflix is continuously growing, and in 2010 with the focus moving to streaming media it started to transform into a digital business.
In 1998 Netflix started with only 30 employees, in 2018 they were 7,100. As of April 2019, Netflix had over 148 million paid subscriptions worldwide.

The two technical components of Netflix’s scale are their CDN and their control panel (2016).
The CDN (Content Delivery Network) runs on Netflix own hardware around the globe, and It’s by far the largest in the world. It handles all of the video content and at the peak, it reaches over a third of the bits on the internet.
The control plane runs in AWS and handles all of subscribers interactions with Netflix service on thousands of devices.
From the engineering point of view in Netflix, there are eighty small engineering teams, instead of one large engineering team.

Netflix has certainly demonstrated its resilience in the face of the dot-com bubble burst, the ability to adapt and innovate its business, the ability to grow and evolve the organisation, and the speed. In a word, Netflix demonstrates Agility.

Netflix’s culture is very adverse to processes. They only hire talented senior engineers and emphasise freedom and responsibility, independent decision-making, high alignment and loose coupling.

They emphasise technical excellence and team-work, in what they call a dream team, that collaborates in a cross-functional way and has ownership of the services and capabilities they build following the “you build it, you run it” approach.

All these aspects of Netflix’s culture are very well aligned with several Agile principles.
Another element of Netflix’s culture is that Netflix keeps only their highly effective people with a mechanism called the “keeper test” implemented by managers. This element has the effect to lower psychological safety that is one of the ingredients that support and enable collaboration, so it is not aligned with the people side of Agile and the Agile principles.

Netflix moves very quickly and most teams iterate in production very regularly. The Agile principle that the best measure of success is working software is definitely present in Netflix.
On the other side teams in Netflix are build around micro-services and not all of them are customer-facing, not all of them need to interact frequently with businesspeople and stakeholders. So the Agile principle that suggests that business people and developers must work together daily, it is not present in Netflix.

Netflix certainly doesn’t explicitly adhere to Agile.
There are some teams in Netflix that subscribe to some Agile frameworks, so Agile exists in pockets at Netflix, but as a whole, enforcing a framework at scale is both unwieldy and counterintuitive to the structure of the individual teams.


Below the suggested content.

Article/Podcast: How Netflix Embraces Complexity without Sacrificing Speed: An Interview with Casey Rosenthal
https://www.stickyminds.com/interview/how-netflix-embraces-complexity-without-sacrificing-speed-interview-casey-rosenthal

Post: Netflix Culture
https://jobs.netflix.com/culture

Article/Podcast: What I Learned About Building High-Performing Teams by Surviving Netflix’s Dot-Com Bubble Burst
https://thinkgrowth.org/what-i-learned-about-building-high-performing-teams-by-surviving-netflixs-dot-com-bubble-burst-7f9e87fdb29f

Video: Refactoring Organizations – A Netflix Study
https://www.youtube.com/watch?v=ymIZ5HhH0o4
https://www.infoq.com/presentations/netflix-refactor-organizations/

The Netflix Tech Blog
https://netflixtechblog.com/

The HP LaserJet Firmware Case Study

The HP LaserJet Firmware division is an example of a large-scale software development initiative that adopted Agile ways of working. That is a very large number of software developers working on the same piece of software

The team initially consisted of 400 people distributed across the USA, Brazil, and India, developing embedded software and working on a codebase with several million lines of code.

They were moving too slowly in relation to the speed of the market and the evolution of the products. As a result, the company’s printer products were lagging behind the competition.

Between 2008 and 2011 they adopted Agile ways of working inspired by the lean/agile principles and with further inspirations from the wide literature on the several official Agile frameworks (among them Scrum and Extreme Programming) and related practices.

They did not use or refer to any commercial scaled agile framework.

There was a huge focus on technical aspects such as test automation, continuous integration and continuous delivery, etc.


Below the suggested content.

Article: The Amazing DevOps Transformation Of The HP LaserJet Firmware Team (Gary Gruver)
https://itrevolution.com/the-amazing-devops-transformation-of-the-hp-laserjet-firmware-team-gary-gruver/

Article: The HP FutureSmart Case Study
https://continuousdelivery.com/evidence-case-studies/#the-hp-futuresmart-case-study

This journey has been extensively documented in the book:
– A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware. Addison-Wesley.

It is also mentioned in the book:
– Lean Enterprise. Jez Humble, Joanne Molesky, and Barry O’Reilly

Thanks

Thanks to Carlo Beschi, Corrado De Sanctis, Michal Vallo, Tony Richards, Zorzis Mimides, and Bülent Duagi for the improvements they suggested.

Thanks to Sheen Yap and Stuart Grimshaw for suggesting some of the sources.

Their feedback does not imply an implicit endorsement to this post.