The way we understand Agile needs to evolve.
Remember that our higher order objective is to validate our ideas the fastest, cheapest way possible. Actually building and launching a product idea is generally the slowest, most expensive way to validate the idea.
- Marty Cagan
Even though organizations are adopting more modern and efficient Agile software development methods (like SCRUM and Kanban), they are still having trouble delivering what customers actually want or need. Although we’ve experienced many benefits to the modern methods and practices, we’re also experiencing a long list of issues as well.
For example:
- Customer or market needs are not fully understood
- Envisioning a big picture is difficult when building incrementally
- Velocity is prioritized over product value
- Fuzzy or missing requirements
- Single voice for backlog prioritization
- Scope Creep
- Buildup of UX and technical debt
- Infrequent collection of customer input and feedback
Misunderstanding Agile
We all know that these and many other complaints about Agile based processes aren’t new. In fact, I’ve had lengthy discussions with various professionals on how it’s “bad” for over a decade now. But from the very beginning, each conversation I’ve had contains the same basic themes on how people view and misunderstand this method:
Theme 1: Agile is fundamentally flawed
The individuals that have had this point of view represent a wide variety of disciplines within product development teams. They typically believe that Agile only advocates delivering solutions and that velocity is the key metric for success.
Theme 2: Agile focuses on development and not the user
Most of the people that have had this viewpoint are research or design professionals. They believe the majority of the activities that are practiced treat research and design as secondary or unnecessary.
Theme 3: Agile and SCRUM are the same things
Almost all of the development professionals I have spoken to believe Agile is SCRUM. They were either taught by a coach or learned on their own about Agile based processes and now have a tight association with understanding Agile and it actually being the SCRUM framework.
For me, all of these misunderstandings sum up to a single issue of not really knowing what Agile is.
So what is Agile…really?
There are a lot of definitions out there and most of them are defining either a set of processes, a workflow or a framework. All of these definitions confuse the intent of what it really is and unfortunately, they keep getting referenced and reinforced.
Here’s the most basic definition: Agile is a set of values and principles
That’s it. Well, sort of. It also helps to understand the values and principles that were originally created:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
To learn more about the values and principles, the original members that created it, and the thousands of people that support it, please visit the Agile Manifesto website.
So, why do people have so many issues and say that Agile is bad? My theory is, everything they’ve probably heard or have been taught is either a specific method, process, or iterative frameworks that support these values and principles. For example:
- Scrum
- Kanban
- Lean
- Adaptive software development (ASD)
- Agile modeling
- Agile Unified Process (AUP)
- Disciplined agile delivery
- Dynamic systems development method (DSDM)
- Extreme programming (XP)
- Feature-driven development (FDD)
- Rapid application development (RAD)
With all of these development methods being taught as Agile instead of these as being Agile, it’s no real surprise that people get frustrated.
Now what?
Beyond helping to spread the word of what Agile really is, we need to conduct an honest retrospective of Agile and its most popular methods, identify and isolate the core problems, define potential improvements, and then implement solutions.
Fortunately, it has already started.
In 2007, only six years after the Agile Manifesto was first created, Desiree Sy published an article describing issues she and her team at Alias (now Autodesk) had experienced when first adopting Agile methodologies. This article also outlines how the team embraced the Agile values and principles, and eventually adapted their processes in order to adopt a development methodology that worked for them.
This article, among others, helped inspire thought leaders like Kent Beck, Jeff Patton, and Marty Cagan to think differently about the method in order to improve the definition and spread the word on how to augment some of the core practices.
What’s Next?
At the Startup Lessons Learned Conference hosted by Eric Ries in 2010, Kent Beck talked about Agile programming and how the Agile Manifesto should evolve for startups. You should check out the video, but here is the basic overview of the update he proposed:
Team vision and discipline over individuals and interactions
Validated learning over working software
Customer discovery over customer collaboration
Initiating change over responding to change
Even though the updated version he proposed was originally meant for startups, I believe these evolved values hold true for mature organizations too.
The Evolution of Agile Methods
Taking the insights from Desiree’s article and years of experience working with and consulting Agile teams, Jeff Patton and Marty Cagan have evangelized an evolution to the core methods and call it Dual Track Scrum; or also known as Dual Track Development, Product Discovery and many others.
To break it down simply, Dual Track Scrum is a parallel process that supports two components (or tracks) inside of a single development model:
- Continuous Discovery
- A process that helps us understand and decide what is valuable*
- Focused on learning and validation
- A process that helps us understand and decide what is valuable*
- Continuous Delivery
- A process that allows us to develop and deliver value*
- Focused on speed
- A process that allows us to develop and deliver value*
(*value is for the customer, company and the user)
These evolutions paired together are powerful and I believe the majority of the issues we all experience with implementing current methods could potentially be resolved if we updated our understanding and implementation.
Download Continuous Discovery Poster
Read more about Continuous Discovery
Summary
Now I know there is no single silver bullet or panacea for creating good products or services. There are a lot of things that need to fall into place, and most of it starts with the culture of the organization. But in the spirit of Lean methodologies, we can choose to learn, measure, and build upon what we already find valuable in order to evolve.
How Handrail Can Help with Continuous Discovery
Incorporating Continuous Discovery into product development means you’ll be doing more research. Handrail speeds up user research by eliminating the busywork of planning, collecting, analyzing and sharing research. Give Handrail a try Free for 30 Days.
Mat Winegarden
Product manager at Handrail. Sometimes I have ideas...other times I am brilliantly late to the party.
Matt, you analysis is thoughtful and thorough. Agile as a set of values and principles is lacking in some critical areas. This lacking is okay as long as everyone understands these limits and attempts to cover the needs through other means. Traditional project and operations management does a better job covering these areas but is often too rigid for the exploration that software development becomes.
An organization running a project needs to make business decisions across its project portfolio. The business must understand risks, issues, cost, scope, duration, and value before it can decide if the investment is worthwhile. An organization will have a tolerance for variance and a willingness to balance the cost of planning with known risks. However, nowhere in the Agile manifesto or any of the many implementation forms does a project or operation managed through an Agile method use a systematized method to look forward and attempt to predict a long term outcome. I have yet to see or experience a good integration of an Agile methodology into a traditional or modern project or operations approach that can satisfy the basic organizational needs listed above.
Agile as a set of values is also vague on how to handle the development and integration of systems of systems. The cross communication and coordination cannot be captured in a scrum or kanban approach, which is typically measuring throughput of logged issues. Efficiency when integrating systems only comes through planning and prediction analysis, with an iterative process for defining a set of actions required for a successful integration. In projects I have managed this process happens outside of the Agile methodology used and then is logged as a set of issues to be handled in the Agile format used.
An Agile approach does a great job of focusing and guiding a development team to deliver value quickly and efficiently. One should not discount the value of the gained efficiency of the development team. However, projects and operations have many more factors that need to be considered than just the efficiency of one of its core teams. Until an Agile approach can integrate well with the other critical needs of a project it will be considered deficient. Perhaps Agile 2.0 is not about extending Agile methodologies but finding ways to integrate other project and operational approaches with Agile methodologies so that an integrated approach meets all the customer’s needs.
Hi Roland - Thanks for your kind words and your wonderful additional thoughts!
I agree with a lot of your points outlining issues and absolutely agree that more traditional methods should be practiced in key areas. It’s great that some groups are already doing this (sometimes humorously known by names like agilfall or watergile) but in my opinion there aren’t enough orgs speaking up on how they learned, adapted, and evolved for their specific needs.
What I would like to see in the future is successful organizations sharing what’s worked for them so we can all learn instead of just hearing about individuals fighting about which method is right.
“However, nowhere in the Agile manifesto or any of the many implementation forms does a project or operation managed through an Agile method use a systematized method to look forward and attempt to predict a long term outcome.”
I don’t see anyplace in the manifesto where it dictates what tools or processes should or shouldn’t be used.
A genuinely (small-a) agile organization does what it needs to do.
Hi Hassan - Great call out with the agile vs. Agile! I think this difference is confused quite a bit.
The fundamental issue with the Agile process, like may processes before it (RAD, JAD, SCRUM, eXtreme, etc.), is assuming that design and development can occur at the same time. Design, which is iterative by nature, should always be performed before development starts in earnest (though some things can be done while design is on going to pipeline the overall process). Good design processes are both iterative and agile, but the development process should NOT be iterative in the UI (or UX), only in the code that produces the UI, the middleware, or the backend in an attempt to optimize the code or to develop a new API. An Agile DEVELOPMENT, using today’s processes, would work fine AS LONG AS THERE IS A DESIGN TO FOLLOW.
Consider this proposal: In an ideal world, the flow between design and development would be a waterfall process. This would show that the design was fully executed and vetted before development begins. Imaging how that would change the current costly, iterative development process.
Hi Bill, thanks for sharing your thoughts! I do agree there should be some shared practices between waterfall and Agile methods, but it really depends on on what the goals are. In my opinion, a lot of groups get too caught up in following a specific practice instead of figuring out what works best for them and the product they are creating. And honestly….some groups should only do waterfall.