What do you do on a project where practical constraints impeded pure agile play – Dump agile and adopt waterfall or any other traditional approach OR improvise to save agile? Faced with such a situation on a project I had just joined, I chose the latter. Doing so not only saved the agile core of the project but also made everyone else more agile with the passage of time. Here’s how it happened.
When All Hell Broke Loose
The first iteration of the project was a disaster. On the last day of the iteration, no piece of software was in working condition. The end iteration checkpoint was postponed, against the practice of time boxing. This was done to avoid the bigger fiasco it would have been, had the end iteration checkpoint been conducted sans a working demo in front of the who’s who of the Client team. When the end iteration checkpoint finally happened, business users were left confused and unhappy with what they called incomplete business processes. They assumed that the development team had not understood the requirements correctly. Escalations were made and all hell broke loose with every stakeholder washing his/her hands off the entire episode.
An informal project rescue effort resulted in identification of the following key impediments to agile that in turn had resulted in the fiasco that the first iteration was.
Process Compliance Needs at the Client end: Working in the banking environment with its regulatory frameworks and strict compliance norms, the client organization worked under strong process control. A single line change to the project requirements had to undergo review by a designated panel of eight to ten people, followed by a formal sign off by the heads of six distinct departments. This made it impossible for a single person to be designated as the Product Owner. The development team was thus obliged to gather requirements from a group of business analysts (BAs) who did not have the sign off authority. With BAs following their standard approval process before giving a go ahead to the development team, almost half of the first iteration was lost in gathering requirements.
Lengthy business process definitions: The application involved complex and inter-twined business processes that could not be broken into smaller processes of business value. At the same, re-usability requirements of the application severely restricted the team’s ability to deploy multiple team members to work on different parts of a process in parallel. The resultant sequential development of long business processes meant that almost none of the business process could fit into the four weeks of the iteration.
With no Product Owner to clarify the real business, in their own wisdom, the team broke down the business processes so as to fit the stories into a single iteration. The resultant part-processes from these stories were incomplete in the eyes of the business users.
Constraints on Technical Architecture: The application under development was to be deployed in an enterprise environment and was expected to exchange data with Third Party Application Programming Interfaces (APIs) of five legacy applications. To validate compatibility, all interface designs required prior review and sign off from the respective legacy application’s architect. This meant that the interface design had to mandatorily precede its development and there was no way the interface implementation could be spliced-up as self-contained user stories.
Background processes: The application had certain common security and audit processes that ran in the background for every business operation. While the effort required to complete development of these background processes was substantial, stories associated with these processes were neither demonstrable, nor did they have any business value of their own unless associated with a front end business operation. Demonstration of front end business operations in the first iteration sans the background processes was amongst the key reasons that made business users assume that the team had not understood the requirements.
The Solution – Improvised Agile
Given how difficult it was in the client organization to alter processes and how sensitive people were to being bypassed in a process, the client stakeholders felt that trying to resolve the above impediments was going to be a challenging and time consuming task that could pose a risk to the application’s planned roadmap. At the same time, the client project management team did not want to move to waterfall and was keen on having an approach where incremental functionality could regularly be delivered to client environments. Therefore, they agreed to a mixed approach that maintained regular delivery of software while compromising on some other agile aspects. This approach was laid on two key principles:
- Maintain the agile core of execution such that incremental functionality is delivered at regular intervals.
- Introduce non-agile or a part-agile components in areas where a pure agile approach could put the project at risk.
With all on board, the following hybrid components were incorporated on the project. Sooner or later, each of these hybrid components became more and more agile.
Requirements Gathering: Akin to waterfall, requirements-gathering was retained as a separate phase during which business analysts were to document the requirements and get them signed off by the designated stakeholders. Not surprisingly, the requirements gathering phase started to stretch and it wasn’t long before all stakeholders ran out of patience. It was then decided that similar to agile, business analysts will create requirements for the incumbent release in order of the business priority of features and functionality. This allowed for the first iteration to be kicked off even as requirements for future iterations were being documented. As the first iteration started, queries started to pour in from the development team. Wary of the delays caused by their regular sign-off-before-confirmation process, the clients designated a group of 3-4 BAs as the de-facto Product Owner. Any clarification from this group of BAs was deemed “signed-off” for the purpose of the development team while the BAs internally managed their bank’s sign-off process with respective stakeholders.
The true value of just-in-time requirements gathering came to the fore when change requests started to pour in. The development team’s openness towards embracing change and zero cost of changes for features not yet implemented went a long way in helping the clients overcome their fear of change requests. Reviewers got comfortable with signing off the requirements based on their present best understanding rather than trying to conjure up every possible future scenario upfront.
Background processes: The development team utilized its capacity during the requirements gathering phase to create mid-level designs and proof-of-concepts for the background processes. This created the framework for background processes and allowed their low level design and coding tasks to be embedded within the linked functional story as a task. The result was that the business users got to see complete business processes when working software was demonstrated.
Lengthy Business Processes: The business processes were analysed to arrive at eight weeks as the optimum time for delivering working features without compromising on the business value. The clients agreed to adoption of an improvisation that required working software to be delivered at the end of what was to be an eight-week Release. Agile practices associated with the Iteration were shifted to the Release level. Scope and user stories were prioritized for a Release and Kick offs, Checkpoints and Retrospectives were conducted at the start and end of a Release.
As the team and clients moved to better understanding of agile tenets, the team started to conduct each release as two distinct iterations of four weeks each. This was done by creating “bucket” user stories spanning the entire release. The bucket stories were then split into multiple smaller, testable stories that were scheduled for the two iterations within a release. End iteration checkpoints were held with a controlled set of client stakeholders who were directly involved in product planning and development. This increased the process agility by facilitating mid-release course correction or introduction of high priority changes at the end of the first iteration within the release.
API Designs: Design stories created for the APIs were scheduled over multiple iterations such that their level of effort was no more than 20% of the total level of effort of the respective iteration. This ensured that 80% of the effort still went into creating demonstrable user stories. As the external applications themselves began to undergo changes to support new or changed business requirements, their corresponding API definitions started to change as well. In no time the clients realized that prior design approval of APIs could get stuck in an endless loop of change requests. The 20-80 LOE rule was then discarded in favour of splicing up the API design to match with the development of core business processes that interacted with the APIs. API design and development were subsequently scheduled as tasks under the user story/stories for the corresponding business operation(s) with API owners providing incremental sign off for implemented designs.
The project was completed on time, with a record margin that exceeded the margin at which the project was sold, by 20%. Passing through the various releases, everyone from the client stakeholders to the project team became more agile and collaborative in areas where non-agile practices had been established. A bit of improvisation thus went a long way in retaining the agile approach that was envisioned for the project.