How large successful companies achieve agility at scale – Part 2
This is an account of how large successful companies like
Microsoft, Amazon, Google, Facebook, Spotify, LinkedIn, Netflix and HP LaserJet Firmware
achieved agility at scale with tangible and lasting business benefits.
<< Go back to Part 1
(Summary, Microsoft, Amazon, Google)
Disclaimer: I don’t and haven’t worked for these companies. The info reported below has been cross-checked from multiple publicly available sources (videos, articles, documents, posts) coming from, then, company’s employees, in few cases from a journalist that visited the company, in one case from a previous employee, and from the company’s blog. For some of the companies, I could double-check with a current employee the resemblance of the whole resulted from putting together info from different sources.
Facebook in 2019 had 43,000 employees and 2.45 billion monthly active users (Source: Facebook 10/30/19). Since its launch in 2004, Facebook is continuously growing, and so their ways of working are continuously changing and evolving.
Facebook is not only a large company, but it is also innovative, fast, flexible and resilient. It demonstrates agility at scale.
The company culture is very engineering-driven. Engineers contribute to defining the direction of the product, or product’s capabilities, they work at, and the related roadmap. Engineers also have degrees of autonomy in their ways of working and the evolution of the ways of working, that is informal and organic.
Facebook to deal with the speed of growth and its scale, has to develop some of its tools. For example, Facebook employs a single company-wide source code repository, a mono repo, that handles thousands of commits a week across hundreds of thousands of files from a codebase of several million lines of code.
In terms of technical practices, Facebook practices Continuous Integration at scale, Test Automation, Trunk Based Development, and Continuous Delivery releasing into production many times per day. Facebook also practices collective code-ownership: any engineer can modify any part of FB’s code base and check-in at-will.
All these technical practices have their roots in Agile Software Development and more specifically in Extreme Programming.
In Facebook, the work is organised in small cross-functional teams that own and are responsible for a product or product capabilities. This approach has been popularised by Agile Software Development. Some teams have been reported to adopt some of the practices from some Agile frameworks (e.g. daily standup).
Facebook does not make any explicit references to Agile in relation to their ways of working, in any of their communications, while taking some inspirations or using the evolutions of Extreme Programming practices.
In relation to scaling their ways of working instead there is no reference at all, no inspiration, no use of any idea or evolutions from any of the commercial agile scaled frameworks.
Below the suggested content.
Product engineering at Facebook (notes on cross-functional teams)
– https://engineering.fb.com/android/product-engineering-at-facebook/
See paragraphs ‘Small, Independent Teams’, and ‘Control and Responsibility’
– https://www.facebook.com/note.php?note_id=409881258919
Facebook’s Trunk-Based Development
– https://paulhammant.com/2013/03/13/facebook-tbd-take-2/
Scaling Mercurial at Facebook
– https://engineering.fb.com/core-data/scaling-mercurial-at-facebook/
Ship early and ship twice as often
– https://engineering.fb.com/uncategorized/ship-early-and-ship-twice-as-often/
See paragraph ‘Continuous Integration testing’
– https://trunkbaseddevelopment.com/vcs-features/
Facebook engineering
– https://engineering.fb.com/
Spotify
Spotify is another well-known example of a large tech company adopting Agile ways of working. They did it while growing very quickly. Spotify was founded in 2006. In 2012 they had hundreds of developers divided into 30 agile teams spread over 4 cities in 3 timezones. In 2014 they had 1500 employees of which 700 in Tech, Product and Design. In 2016, 2000+ employees.
To be Agile at scale they did not use any commercial scaled agile frameworks.
They grow their ways of working internally. In this ongoing journey, they have been inspired by Lean and Agile principles among other things. Teams used Scrum, Kanban, and Lean Startup. And teams had a strong focus on the engineering aspects. Some of the Agile technical practices (such as TDD, Pair programming and Simple design) were used by some teams but not from other, other Agile technical practices (like Continuous Integration, Continuous Delivery and DevOps for example) were more widespread.
Spotify was and is not an Agile nirvana. For example in the squads (the teams) there was an imbalance between the decision right of the Product Owner and the technical counterpart leading to technical debt for some teams. The allocation of some line management responsibilities outside the teams lead to a matrix management structure with the well-known downsides. Some teams become too specialised (as opposed to owning end-to-end a product area or function) leading to many cross-team dependencies when dealing with bigger features. Agile Coaches were stretched too thin not giving enough support to teams to learn how to align and how to collaborate in the face of cross-teams dependencies.
Between 2012 and 2014 Spotify shared a simplified description of their ways of working and the several recurring “patterns” they have observed there. Those descriptions were a snapshot at that specific time of a quickly and always evolving company. They have never been intended to be used as a model to be replicated internally for every corner of the organisation (no cookie-cutters) or to be imitated or copied externally by other companies (no copy-and-paste). Because that is fundamentally in contrast with how agile ways of working grow and evolve.
Below the suggested content.
Pdf: Scaling Agile at Spotify
– https://blog.crisp.se/2012/11/14/henrikkniberg/scaling-agile-at-spotify
Video: Spotify Engineering culture
– https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/
– https://labs.spotify.com/2014/09/20/spotify-engineering-culture-part-2/
Challenges at Spotify
– https://www.jeremiahlee.com/posts/failed-squad-goals/
LinkedIn was founded in 2002 and launched in 2003. As reported in 2016, the company has around 9,200 employees with 33 global offices.
Starting from 2011 LinkedIn switched from a waterfall-like development process to modern ways of working, going from a system that requires a month to release new features to one that pushes updates into production multiple times per day. Kevin Scott, the senior vice president of engineering, and his team of programmers had a key role in this switch.
LinkedIn’s big switch has been linked to very concrete and visible financial success, and business capabilities.
LinkedIn’s new and modern ways of working revolve around Continuous Delivery, with Test Automation, and Trunk Based Development. All these practices have their roots in Agile software development and more specifically in the Extreme Programming Agile framework. LinkedIn indeed takes inspiration from Agile frameworks, such as Extreme Programming (Scrum and Kanban are also mentioned in some of their job postings), while it does not formally adopt a specific Agile framework.
Furthermore in LinkedIn’s ways of working there is no reference at all, no inspiration, no use of any idea or evolutions from any of the commercial agile scaled frameworks.
In LinkedIn, there is a strong Engineering culture and a culture of tech excellence. With teams having degrees of autonomy on their ways of working.
LinkedIn’s new ways of working required a huge commitment from both the business and the tech, at some point the process required halting all development for two months as LinkedIn trained staff, migrated old code, and built out their own automated tools and infrastructure needed to make the new system work.
The benefits for the business have been immediately visible.
Examples of features that have been quickly created and released thanks to LinkedIn’s new and modern ways of working, include revamped company pages, revamped profile pages, overhauled notifications, a redesigned homepage, comments and likes on news pages, “people you should hire” suggestion box for recruiters, mobile apps, a job listings app, blogging, … and the list goes on. Other feature would have been simply impossible to create without LinkedIn transforming his ways of working, examples include endorsements, the new influencers product, the new version of profile and the several things in mobile.
Below the suggested content.
Article: The Software Revolution Behind LinkedIn’s Gushing Profits
– https://www.wired.com/2013/04/linkedin-software-revolution/
Post: Engineering Culture at LinkedIn
– https://engineering.linkedin.com/engineering-culture/engineering-culture-linkedin
Post: Scaling Decision-Making Across Teams within LinkedIn Engineering
– https://engineering.linkedin.com/blog/2018/03/scaling-decision-making-across-teams-within-linkedin-engineering
Posts: LinkedIn Engineering blog etc
– https://engineering.linkedin.com/
Post: How Agile Helps the GTS Team Get Laser Focus
– https://engineering.linkedin.com/blog/2015/11/how-agile-helps-the-gts-team-get-laser-focus
Netflix
Netflix was launched in 1998 with a focus on the initial DVD rental business. It is in 2010 when Netflix expanded its business with the introduction of streaming media, and internationally. In 2012 entered the content-production industry.
Since after the dot-com bubble burst in 2000, Netflix is continuously growing, and in 2010 with the focus moving to streaming media it started to transform into a digital business.
In 1998 Netflix started with only 30 employees, in 2018 they were 7,100. As of April 2019, Netflix had over 148 million paid subscriptions worldwide.
The two technical components of Netflix’s scale are their CDN and their control panel (2016).
The CDN (Content Delivery Network) runs on Netflix own hardware around the globe, and It’s by far the largest in the world. It handles all of the video content and at the peak, it reaches over a third of the bits on the internet.
The control plane runs in AWS and handles all of subscribers interactions with Netflix service on thousands of devices.
From the engineering point of view in Netflix, there are eighty small engineering teams, instead of one large engineering team.
Netflix has certainly demonstrated its resilience in the face of the dot-com bubble burst, the ability to adapt and innovate its business, the ability to grow and evolve the organisation, and the speed. In a word, Netflix demonstrates Agility.
Netflix’s culture is very adverse to processes. They only hire talented senior engineers and emphasise freedom and responsibility, independent decision-making, high alignment and loose coupling.
They emphasise technical excellence and team-work, in what they call a dream team, that collaborates in a cross-functional way and has ownership of the services and capabilities they build following the “you build it, you run it” approach.
All these aspects of Netflix’s culture are very well aligned with several Agile principles.
Another element of Netflix’s culture is that Netflix keeps only their highly effective people with a mechanism called the “keeper test” implemented by managers. This element has the effect to lower psychological safety that is one of the ingredients that support and enable collaboration, so it is not aligned with the people side of Agile and the Agile principles.
Netflix moves very quickly and most teams iterate in production very regularly. The Agile principle that the best measure of success is working software is definitely present in Netflix.
On the other side teams in Netflix are build around micro-services and not all of them are customer-facing, not all of them need to interact frequently with businesspeople and stakeholders. So the Agile principle that suggests that business people and developers must work together daily, it is not present in Netflix.
Netflix certainly doesn’t explicitly adhere to Agile.
There are some teams in Netflix that subscribe to some Agile frameworks, so Agile exists in pockets at Netflix, but as a whole, enforcing a framework at scale is both unwieldy and counterintuitive to the structure of the individual teams.
Below the suggested content.
Article/Podcast: How Netflix Embraces Complexity without Sacrificing Speed: An Interview with Casey Rosenthal
– https://www.stickyminds.com/interview/how-netflix-embraces-complexity-without-sacrificing-speed-interview-casey-rosenthal
Post: Netflix Culture
– https://jobs.netflix.com/culture
Article/Podcast: What I Learned About Building High-Performing Teams by Surviving Netflix’s Dot-Com Bubble Burst
– https://thinkgrowth.org/what-i-learned-about-building-high-performing-teams-by-surviving-netflixs-dot-com-bubble-burst-7f9e87fdb29f
Video: Refactoring Organizations – A Netflix Study
– https://www.youtube.com/watch?v=ymIZ5HhH0o4
– https://www.infoq.com/presentations/netflix-refactor-organizations/
The Netflix Tech Blog
– https://netflixtechblog.com/
The HP LaserJet Firmware Case Study
The HP LaserJet Firmware division is an example of a large-scale software development initiative that adopted Agile ways of working. That is a very large number of software developers working on the same piece of software
The team initially consisted of 400 people distributed across the USA, Brazil, and India, developing embedded software and working on a codebase with several million lines of code.
They were moving too slowly in relation to the speed of the market and the evolution of the products. As a result, the company’s printer products were lagging behind the competition.
Between 2008 and 2011 they adopted Agile ways of working inspired by the lean/agile principles and with further inspirations from the wide literature on the several official Agile frameworks (among them Scrum and Extreme Programming) and related practices.
They did not use or refer to any commercial scaled agile framework.
There was a huge focus on technical aspects such as test automation, continuous integration and continuous delivery, etc.
Below the suggested content.
Article: The Amazing DevOps Transformation Of The HP LaserJet Firmware Team (Gary Gruver)
– https://itrevolution.com/the-amazing-devops-transformation-of-the-hp-laserjet-firmware-team-gary-gruver/
Article: The HP FutureSmart Case Study
– https://continuousdelivery.com/evidence-case-studies/#the-hp-futuresmart-case-study
This journey has been extensively documented in the book:
– A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware. Addison-Wesley.
It is also mentioned in the book:
– Lean Enterprise. Jez Humble, Joanne Molesky, and Barry O’Reilly
Thanks
Thanks to Carlo Beschi, Corrado De Sanctis, Michal Vallo, Tony Richards, Zorzis Mimides, and Bülent Duagi for the improvements they suggested.
Thanks to Sheen Yap and Stuart Grimshaw for suggesting some of the sources.
Their feedback does not imply an implicit endorsement to this post.
You may also like:
– Agile at scale generative principles
– Agile practices & patterns for the whole Organisation
– Case studies of unsuccessful adoptions of Agile at scale
If you’ve made it this far then you’re probably interested also in:
– Executive summary for everyone considering investing in Agile: https://bit.ly/AgileInvestingExecutiveSummary
– Information for decision-makers considering the SAFe framework: https://bit.ly/SAFe4DecisionMakers
– Information for decision-makers considering large consultancy firms’ Agile offering: https://bit.ly/agileOffering4DecisionMakers
Turbocharge your scaling initiative.
See how we can help.
You, your team, your organisation.