Automation practices in Agile Software Development have an important role. But what about Agile outside IT? What to automate? How to automate?
In all successful organisations and effective teams I’ve been part of while adopting Agile Software Development, automation practices always played a fundamental role. Both in terms of speed, quality, and safety.
Those that immediately come to my mind are Automated Testing, Continuous Integration (with Trunk Based Development), Continuous Delivery (with automated remediation plans), and DevOps.
Automation practices are so important that even when Lean and/or Agile are adopted outside IT, their fundamental benefit cannot be ignored.
Good candidates for automation include standard work, repetitive tasks, and defined processes that occur often, starting from those that are more expensive, require more time and effort, and are more error-prone.
Common examples include various forms of assembling and integrating, testing and reporting, as well as data analysis and monitoring. Also, data collection and integration are good examples (e.g. turning them from manual operations to real-time telemetry).
All the tasks with a high degree of novelty (may involve creating new solutions, designing new things), and in general those that require human creativity and ingenuity, imagination, and intuition, are not good candidates for automation. Nor are those tasks that involve exploring, experimenting, discovering, and guessing.
Also, all the tasks that involve making sense of things collectively, building and reaching consensus and taking decisions, dealing with human beliefs, and dealing with power or emotions, are not good candidates for automation either.
All these tasks deeply involve what I call the human element, and that is why for them a real person needs to be in control, be in the loop, and make the final call.
Step 1 – Documentation
Once a task that is a good candidate for automation is found (usually a repetitive, time-consuming, low-variation task), the first step is to document it for someone else to carry out the task manually, following the documentation.
This will allow someone with limited knowledge or context of the details of the task, to carry out the task anyway. You can think for example about the difference between someone knowing how to create a formula and its proof Vs someone knowing only how to apply a given formula. The latter is all that should be needed to follow the documentation and carry out the task.
The tool used to document the steps should be like a Wiki, a tool that doesn’t make a distinction between the author and the readers, and everyone is an active reader (both author and reader at the same time). Those using the documentation to carry out the task will ask questions to fill the gaps and to clarify any ambiguity in the documentation, and so evolving, updating and improving it.
It is important that this documentation is valuable in itself and it is actually used and useful. This will lead to improving the documentation and will prove that there is a need for help to carry out the task and that the delegation of the task to others actually works. And so a trail of changes to the documentation will be the sign that this first step has been valuable and successful.
Step 2 – Scripting (checklist / flow diagram)
The next step is to streamline the documentation, making the instructions simpler and more straightforward. Eliminating any part that can be subjective. Providing clear unequivocal criteria for every decision that needs to be taken. Bringing down almost to zero the level of interpretation and the judgment calls that the reader needs to make in order to carry out the task. Providing clear instructions also to deal with error conditions that can happen on the way, as well as to deal with variations that there may be in the inputs to the task.
This format of this streamlined documentation is much easier to follow, leaves much less room for error, and requires much less knowledge from the reader. So if it really works, a larger group of people will be able to follow its instructions.
As before, it is important that this documentation is valuable in itself and it is actually used and useful. This will prove that there are not too many variations in the task that make it not a good fit to be automated. And that the instructions to carry out the task have an actual chance to be automated effectively (by an algorithm, a smart robot with physical abilities, or basic cognitive abilities that can be trained into commercially available AI).
Step 3 – The actual automation
After the scripted documentation from the previous step has been proven to work well repeatedly, without tweaks or non-scripted interventions, then it is time for the actual automation.
The process of automation often requires that part of the resources (raw material, or artefacts) used in carrying out the task follow some process of standardisation to reduce variation and make the automation possible and easier.
This step is proof that the automation is actually feasible, that it works well, that is stable and sustainable, and that is beneficial and effective. The automation should be done by those initially responsible for carrying out the task, the experts. Their understanding and knowledge are fundamental in judging the solidity of automation and its sustainability.
Because the work we do and the way we work is in constant evolution, also the automated tasks will evolve over time, this is why the skills to maintain and evolve the automation introduced should be embedded and available inside the team using the automation. There are cases where the automation can be standardised and reused elsewhere, this should be proven by the fact that once the automation is done, no more updates to the automation are needed after then. This is where the traditional approach of doing big-bang automation with standard tools bought off-the-shelf typically fails.
The smaller the initial task to be automated, the faster this step is reached, and the sooner it can be determined if it really can be automated successfully. After that, and only after that, the automation can be extended to automate a larger task.
Paolo Sammicheli author of Scrum for Hardware came up with this metaphor.
Think about software developers, they work on a problem, then they translate their solution into source code written in a programming language, and that source code is consumed by a Compiler that turns it into a software application that can run on your smartphone or your laptop.
Steps #1 and #2 above are like when software developers define and translate a solution into source code, which is like a blueprint.
Step #3 is like when the compiler turns that source code (the blueprint) into a software application like the apps in your smartphone that can run automatically with one touch.
Paolo also made a second example, when a CAD designer thinks about building something and then turn that thinking into a drawing of a mechanical part on the computer screen (this digital drawing is in a sense similar to the source code of the previous example). The drawing then is consumed by Computer-aided manufacturing (CAM) machinery that turns it into production steps that create the part.
Steps #1 and #2 above are like when the CAD designer thinks and then draws the part at the computer.
Step #3 is like when the CAM machinery, like a CNC machine or 3D printer, turns that drawing into an actual part.
What brings together #1-2 & #3 in one example is a programming language and its compiler, in the other is a CAD and a 3D printer. Things that you can buy. When automating something for the 1st time, you may or may not find the equivalent tools that you need, so you may have to build them gradually reducing your initial scope for the automation and expanding it as the tools get developed more.
Before Agile Software Development, automation was often approached with a big-bang top-down large initiative. Or trying to buy silver bullet tools promising miraculous results.
The new approach to automation of Agile Software Development has been a huge leap forward, and very successful. Automation has been driven and created by those doing the work. Starting small and gradually evolving custom-made solutions. The role of the tools has been that of a supporting role, leaving to those doing the work the protagonist role they need to make it work.
Here you can find my suggestions on Agile technical practices outside IT, that are relevant also for automation: Agile Technical Practices outside IT.
You will find in the market plenty of tools sold as a silver bullet that usually omit that the biggest cost is creating, maintaining and evolving the automation. And it is very unlikely that such tools will allow for gradual automation driven by those doing the actual work (that is a key element for this new Agile approach to automation that actually works).
To get it right you want to approach the automation bottom up. First, you need someone in your team that is ambidextrous, a tinkerer, someone for example who not only is good with Excel but that also masters Excel macros and Excel scripting. Or the equivalent for the domain you are working in.
Then you have to allow a small ready-available budget and time from the team to find simple solutions to the challenge of automation.
If after this you still need external support to create the automation, remember to clearly state the fundamental constraint that the solution needs to be maintained and evolved autonomously by those doing the work. And the only way to guarantee that it is not just wishful thinking is to have those doing the work create and evolve the first automation and a few more before the automation tool goal can be claimed to be achieved.
Common reasons for task automation to fail are: some part of the task is too hard/expensive to automate; human judgment cannot be removed from the execution of the task; variations in the task inputs or outputs cannot be removed without losing too much value.
In all the other cases, automation will provide a significant increase in speed, a significant reduction in human errors, and a significant increase in the volume of work that can be carried out. We are talking about several orders of magnitude.
The automation process should be approached gradually, going through the three steps described above, starting with a small task, and incrementally increasing the size/number of tasks automated.
The automation should be maintained and evolved by those using it.
To Paolo Sammicheli, Claudio Sauring, Joe Justice and Carlo Volpi for taking the time to discuss this topic with me, share their insights and experience with me, and give me valuable feedback.