“To say ‘I did’, you must start with ‘I am’, ‘I can’, and ‘I will’.”
— Kurian Mathew Tharakan
Before embarking upon the transformation to Continuous Delivery, it may be advisable to carry out a capability assessment, in order to understand the scope of the changes necessary and the sequence in which you should tackle these activities.
It helps to keep in mind that in many cases, you are asking your team to turn their world inside out. They feel that they have a really strong grasp upon the intricacies of a given field, and you are going to ask them to see it again, from a new angle, as if for the first time. Some people will love that opportunity and leap in with glee, but many will be scared, suspicious and all too keen to flee back to familiar old habits at the first excuse. Because of this, it is generally unwise to try and perform an organization-wide transformation to Continuous Delivery as a ‘big bang’ rollout as you will risk triggering ‘corporate anti-bodies’ into play to return the status quo by undermining the change.
Instead, you might consider spinning up a new org unit, with the remit of creating a new product line, or entering a new market segment. This should then be staffed with an executive team who are fully bought into Continuous Delivery as an approach and who are empowered to act as scouts, breaking new ground and finding the path for the rest of the organization to follow, subsequently. This then gives you the freedom to set up new incentive structures for staff within this org unit which are better aligned to the methodology and the product goals. It is recommended to use this opportunity to bring in some new team members who have prior Continuous Delivery experience and set them up as mentors to the rest of the team. You can then start to bring across small numbers of staff from the parent organization and retrain them as they join the new unit. The first and most important goal is to create a culture within this new org unit that is properly aligned to the methodology, and to clearly weed out behaviors that tend to regress to old ways of thinking.
Let’s look at an example in order to better understand some of the patterns of change that are needed:
Many organizations are familiar with a way of working that involves passing a set of requirements to a development team, then accepting back a deliverable which the developers assert is ‘finished’. This candidate product is then passed around a number of departments to perform manual activities to ‘find things wrong with it’. This typically results in an iterative period of remediation and new inspections until enough people feel confident in the product to release it to customers. This rear-end loaded risk management approach adds weeks or months of delays to each release, and still cannot catch everything that could go wrong. Worse, it leaves the customer entirely out of the loop.
In practice, this approach is so slow and expensive that shortcuts are put in place to work around it. Here is a common example. The legal team managing the IP inherent in your product need to understand what components and libraries are being used to implement the product. If those libraries are commercial, does the business have appropriate licensing agreements in place to allow this usage? If the libraries are Open Source, are they licensed in such a way as to permit commercial use, or do they contain clauses that might require you to Open Source your own IP as a result of using them? This requires someone on the development team to trawl through every dependency (and all of its dependencies) to compile a list of applicable licenses. Then someone on the legal team needs to review those licenses and sign off on them as appropriate and properly managed. Run rear-loaded, this is such a laborious and mind-numbing process that it becomes a box-ticking activity that gets done once during the lifetime of the product and then everyone just pretends that new dependencies are not being added daily as the product continues to evolve.
To re-implement this as a Continuous Delivery capability, you have to invert the process and front-load it. The legal team needs to develop policy rules that generically describe classes of licenses to accept or block. The engineering team is then required to provide a Software Bill Of Materials that includes licensing information for every component, against every release. The final part of the capability is a release pipeline activity that scans the SBOM for policy violations. In this way, license management becomes a development activity because it is a prerequisite to get your code through the build process. Your legal team are then restructured to focus on rapid handling of exception processes (dealing with new license types, commercial license management etc.) so that developers can get what they need to build product, at the pace that they need to meet the business drivers.
Once you see this pattern, you will recognize it everywhere in large organizations and the solution is always to find those rear-loaded governance processes and front-load them with automation to free people up to handle the messy edge cases that will inevitably occur.
Before you rush into transformation, however, there is another key element that needs to be in place as early as possible.
Continuous Delivery replaces the ‘inevitably futile culture of expending significant effort to try and stop bad things from ever happening’ with a more pragmatic expectation that ’the unanticipated will always happen’ and the ability to react to it extremely rapidly and effectively. This requires that you put in place processes to react to failures in production as a priority, and that you transition away from a ‘blame culture’ to a ‘continuous learning’ culture where each failure is treated as a positive learning experience to help you improve your systems.
The end goal is to have automated infrastructure and deployment processes that can detect failures and reverse out changes to a last known good condition. If you are starting with a greenfield project, you can jump straight to this point by selecting appropriate tooling, but for most organizations, it will be necessary to develop a series of transitional architectures with manual intervention processes as a roadmap to reaching this goal. Do this early in the process and practice it regularly so that you can implement it smoothly when things do go wrong. The better you prepare in this area, the smoother the rest of your transition to Continuous Delivery will feel, because you have safety nets in place for every change.
An important part of your business change process will be to create and operate incident response teams that are supported with resources from each of the Viewpoints previously discussed. This has significant cultural as well as safety benefits. In your legacy operating model, it is very easy for ‘us and them’ divides to arise. Development teams are typically incentivized to improve quality and are harmed by their own actions if quality drops, but ‘gatekeeping’ teams performing manual reviews on the product rarely have skin in the game so can become perceived as ‘armchair referees’ rather than team players. It is useful therefore to have your supporting business teams embedded within the development process, creating policy to automate governance; AND to have those same individuals on incident response teams, so that they get first hand experience of managing potential policy failures. This creates a much more collegiate environment where everyone has similar skin in the game and everyone gets a better learning experience about what matters most.
It is advisable to temper your ambitions initially and instead create a capability review based upon realistic and reasonable expectations. Teams will need a lot of mentoring as they transition to new ways of working, so prioritize those business functions that perform the most critical governance activities for your product and start there, with a roadmap for other functions to transition as you get more practiced in operation. Remember that a key tenet of Continuous Delivery is creating an environment in which you can make lots of small mistakes that teach you what you need to know to succeed, but which also protects you from making a large mistake that can kill you. One of your first tasks will be to get people used to making mistakes and being praised for turning them into valuable knowledge. If a blame culture creeps back into your new org unit, things are likely to become political and your whole mission may be sabotaged. Plan your capability rollout in bite-sized chunks that allow your executive team to mentor and lead at a comfortable pace initially, until you have sufficient good habits in place to turn up the wick and really start to realize the productivity gains of Continuous Delivery.
As your new org unit becomes more practiced, start to transition more staff across from the parent organization and scale up, making sure to train people in the culture of the new unit as well as the process.