The Elusive Search for Organizational Agile

Even today, most organizations struggle with becoming “more agile.” Paradoxically, some of our failures as managers make Good Agile less likely to happen, and the way many choose to implement Agile creates confusion and failure.

I’m amazed at how poorly the new management fad of Agile is being implemented. My business is cleaning up where others have failed with Agile, so maybe I should not be so surprised, but the scale of it is truly shocking. Most recently, we’ve come into a Fortune 500-sized company that is still reeling from a poorly-designed Agile Transformation that has left many of its teams struggling to make sense of what they should do and not do…all at the hands of one of the most prestigious global consultancies. Beautiful slogans, PowerPoint decks, posters, everything you could want…except that it doesn’t work.

Agile’s not exactly new, though; its first days were almost 20 years ago. So then why is Good Agile, an Agile that really works, so rare these days? It is more like a UFO sighting than a visit by a familiar friend.

It’s dark-sheep cousin, Feckless Agile, seems rampant these days. (I was concerned about this last year after the HBR cover story on Agile, and even wrote about it.) Like any initiative, an Agile transformation’s early signs are often promising — as people enjoy a change in rituals for a while — but in the end they never experience any profound results, and sometimes things get worse.

When I say Good Agile, I mean 20–50% velocity gains, far less managing needed, and startling improvements in collaboration, quality and engagement. Most organizations today struggle to just do a stand-up that doesn’t draw complaints and low attendance. If you have that problem, then you’re living with Feckless Agile.

This epidemic of mediocrity — actually worse than mediocre — comes from a widespread misunderstanding of what it takes to make Agile really work. Virtually every one of these Feckless (or worse) Agile implementations focuses on teaching basic Agile techniques, like the standup, as if these techniques are the magic of Agile — they are not — and ignores many of the key components of any behavioral/organizational transformation, including managerial behaviors and contextualization of the new processes.

This result is that many organizations waste massive amounts of time and money accomplishing nothing except annoying the very people whom Agile should benefit, the teams and workers that actually need to get stuff done. I’m going to explain the phenomenon in more detail, prove to you that basic Agile techniques need little, if any, actual teaching, and then explore several of the key factors that need to be managed correctly in an Agile transformation, all of which are notoriously absent in today’s Agile-as-a-Management-Fad.

Close Encounters of the Agile Kind

Most organizations that try Agile have a success story. Or a few. But they don’t have a lot of them, and they end up being the exception rather than the rule. Let’s look at two very common types of situations:

· Type #1: A very small department (like 5–8 people) or a small, cross-discipline project team, where everyone is dedicated to the project and they try Agile techniques…and they work really well! This is often a pilot project for the organization.

· Type #2: Normal-sized departments, matrix organizations and other “messy” models that are highly functional at accomplishing business as usual activities (a mix of project and department work, often multi-allocated and with fluid priorities)…and the Agile techniques don’t fit, they fail or seem awkward.

The type #1 situation is very close to the dedicated team model that Agile was originally designed for, and as a result the techniques feel somewhat effortless. There are other important attributes to this situation that we’ll explore later.

But type #2, the multi-project work model with fluid and overlapping teams, is a bit crazy and chaotic, maybe endearing and challenging in that way, but pretty much what most departments or organizations look like in the real world. We call it the naturally-occurring chaotic organization. Try Agile in one of these settings, and you’ll learn how hard it can be to get right.

The big consultants often have an answer for this: make everything into a 5–10 person team. There are a lot of problems with this, the largest is that it destroys all of the benefits that accrue to having a department, like lateral skills growth, resource sharing, lessons learned, etc. We saw one implementation designed by a big consultant where they blew up a marketing organization into a bunch of small teams, resulting in the need to hire 15 more managers. If you really know Agile, you know how utterly wrong this model must be; Good Agile, because of its empowered team self-management, results in the need for less managers and managing.

More common, though, is that type #2 failures create the organizational and industrial-scale tragedy like I described in Will Management Succeed at Killing Agile? Confusion abounds. In some moments Agile works really well, and most of the time, it doesn’t. And when it doesn’t work, it creates disbelievers. Word of these failures spreads and negative information seems to always win. As enthusiasm for Agile wanes, others in the organization become more tentative, stop believing or feel empowered to take shortcuts, like not having stand-ups, or well,…just not doing it at all.

In that environment, Agile-that-works becomes like a UFO sighting — an otherworldly moment that a select few people claim, with breathless reports of amazement, to have experienced. Everyone else, skeptical because of the tin-foil hat failures they have seen, dismiss these accounts and their little green men.

A truce occurs within the organization: the consultants and coaches are sent home, the “myths of Good Agile” are allowed to live on, in favor of everyone getting along. But the old ways of working return, more firmly entrenched than ever.

This is when we usually walk in. All momentum lost, goodwill exhausted, and people want to just go back to getting shit done. Agile carnage.

Ironically, the things they worked so hard on, like getting people to do stand-ups, were not the problem…in fact, they required no training at all.

Innately Agile — Not So Alien After All

The misunderstanding of Agile runs deep, even amongst the legions of consultants and practitioners. Coaches and trainers come in and myopically assume that if only everyone knew how to do an “X” (like a stand-up, or a backlog, etc.) properly, all would be right. Myopic because, paradoxically, everyone already knew those drills, had lived them multiple times, except for maybe the coaches themselves hadn’t.

Let’s consider a third type of situation that an organization encounters with some regularity:

· Type #3: The project is on fire. Things are not working, work is not being delivered, people are arguing about the priorities and the Sh*t is hitting the fan in a big way. A good project manager (PM) or other manager steps in and orchestrates the “save” by implementing a handful of really effective techniques that everyone immediately adopts without a second of training.

This is fixable, and you’ve seen it before, whether you’ve done it, or had it done for you. Here’s my circa 1984 playbook of 14 great project management techniques for when a project goes wrong, or at least a subset of them. 1984 was seventeen years before the word Agile was first uttered.

• Call an All-Hands Meeting and Diagnose Failed Assumptions & Process. I would get everyone together to make sure we all understand exactly what is going on, who is doing what, and what’s getting done and what is stuck, broken and confusing. One of the big questions I would ask at crisis-time is “What did we not plan for, or what went wrong?”along with the corollary, “What should we be doing differently?” Yes, this is Agile’s Retrospective meeting; it focuses on both the work, and the process.
• Prioritize and Sequence the Work. Identify the “Musts” and “Nows”. Often naïve planning has resulted in an avalanche of failures, where one schedule failure begets other schedule failures. A better approach is to step back and just identify all of the things that could be done, and then make choices about what really matters, what the priorities are. This is Agile’s Backlog Creation and Grooming, where the team (and customer) agree on rank-ordering of the list of work items.
• Set Near-Term Attainable Goals. Teams work best in short horizons and will often feel overwhelmed by complex schedule questions, like “When will you be done?” When things have been bad, nobody could never answer that question, so I chose a simpler one, “What can you guys get done by the end of the week?” In doing so, a PM creates a time-based work period almost exactly like Agile’s Sprint.
• Reduce Interruptions and Thrashing. Often the Type #3 crisis is amplified by multi-allocated people — people that work on several projects at the same time — who are not getting enough “focus” time on any single project, aka “thrashing.” Agile “locks” the sprint in order to help the team focus.
• Hold Daily Status Meetings, Identifying Needed Support. As the project begins to stabilize, the best PM move is to keep the heat turned up by having daily…wait for it…Stand-up meetings. Yes, not invented by the Agile folks, this technique has been effective since the cave-man era.

Yes, these are what things that we do innately, when things get crazy. You know Agile techniques…we all know the techniques. And when I used these techniques to great success almost 25 years ago, I never trained anyone.

The Paradox of Organizational Agile and Management

So, we have a paradox:

· Agile techniques are quite natural and work well in very simple settings and also in very problematic settings.

· Yet, Agile techniques are very hard to do in day-to-day operations.

The Paradox has a very simple explanation: there exists a set of managerial behaviors that block the natural, innate expression of Agile. You could call them managerial mis-behaviors, but I call them failures. Since our societal notion of managing is so flawed, they are not at all obvious.

Having had the benefit of learning how to make Agile work in really chaotic organizations (advertising and marketing agencies, the epitome of the NOCO-style organization) I learned to see patterns of bad management practice that go unnoticed by most people…in fact, managers seem largely oblivious. Hell, I was oblivious back then as well. Here’s a partial list of the managerial failures that keep Good Agile from happening naturally:

· Managers fail to ensure understanding of the work. Often managers are too busy or lack the techniques needed to properly transfer their understanding of what the stakeholder really needs and why. This leaves teams often learning through trial and error, which is costly in time and effort. Often, estimates, if they exist, are horribly naïve and misleading. Agile planning is pretty much useless in this endeavor for some technical reasons…but few practitioners realize that.

· Insufficient focus on planning and pre-coordination. Planning is valuable and is less costly up front for pretty much every type of work — even when the plan contains many uncertainties, the act of planning makes for better execution; knowing the uncertainties up front often reduces their impact. Just like in the type #3 situation, organizations often plan only very important projects without realizing that any poorly-planned project, even small-ish ones, can destroy a lot of other projects through what we call contagion.

· Failure to precisely define priorities and force tradeoffs. This is one of the most severe failures, and can often be seen as long lists of “#1 priorities” — effectively, everything is important. That’s not prioritizing, that’s avoiding the hard part. You’ll notice that type #1 & #3 situations do not suffer from a lack of priority definition. Managers are responsible for being explicit, rigorous and most important, unified about priority; each should answer the same as any of them. One voice.

· Failure to minimize “thrashing” when people are multi-allocated. Thrashing is when someone gets pulled from one piece of work to another without regard, and in fact, usually to the complete detriment of, productivity. In multi-allocated organizations, multiple managers with unresolved competing priorities will thrash the teams with their requests, leaving it to team members to resolve conflicts. Resolving conflicts, of course, is the job of managers as well.

· Failure to ensure information alignment throughout their teams. Group conversations are ridiculously effective. What managers don’t realize is that chaotic information flows, which come from poorly defined work, competing priorities, unknown work status, and siloed information are far more disruptive than a group discussion. Group discussions, when done well, align people and make them more productive. Simple meetings often fail at this because of their “push” style.

· Reduce the productivity and motivation-sapping effects of managerial dominance. Managers generally do not realize how profoundly their behaviors influence engagement and inclusion (and other factors) within a team. The simple act of asking versus telling has profound consequences upon a team’s ability to take ownership of a topic. Managers confuse the true need for authority with the flawed ideas of dominance and control. (authority versus control, explained here)

Both the type #1 and #3 situations have natural characteristics that ameliorate the managerial failures listed above, rendering managers largely not needed (type #1) or focused on managing precisely (type #3).

Good Agile was Never About Teaching

Virtually every technique that is today associated with Agile existed for decades or even millennia prior to the coining of that word. The techniques are good ways for people to do things, yes, and they are also very natural. The real challenge lies in something that few speak of: contextualization.

What most people don’t realize is the original Agile, Software Agile, took months to get working properly within a team. The coaching process included some techniques education, but more important, it focused on the micro-adjustment of the techniques to fit the specific types and style of work that the team was doing. It could take at least one month, often three or more months of making adjustments every day.

I can understand that you might be skeptical reading this. Because as humans we are so good at “scripted behavior,” embedding dozens or hundreds of behavioral rules into scripts that we execute almost unconsciously, we generally don’t notice how precisely we any given technique. Next time you’re standing in line for coffee, say at a Starbucks, notice how each person is very aware, in an unconscious way, around how they queue in the line, how much distance they seek to keep between themselves and the people around them. Notice how conscious you become when they violate the norms of your script by cutting into line or not advancing fast enough.

That same precision exists in virtually every workplace interaction that we have — we’re just used to them, so we don’t notice them. That’s one big reason why organizational process change is so hard to implement. If you introduce a new process, a new technique, all of those small behaviors need to change, the techniques need to be contextualized to the tasks at hand, and that takes work.

In Software Agile this would be done by a seasoned coach (your recently-anointed scrum master knows nothing about how to do this, BTW) who, understanding how to contextualize Agile for a given business situation, and then manages, mentors and coaches the hundreds of small behavioral shifts that must take place to form a new cohesive and effective process.

Good Agile happens when the techniques have been contextualized.

When you realize this, it can be shocking. Most Agile providers (training and software companies) spend heavily in marketing that aims to (misleadingly) convince everyone that Agile is really easy to do. What chance does a manager with a copy of Sutherland’s Scrum: The Art of Doing Twice the Work in Half the Time, have of getting this right? Pretty much zero. Of the 100-plus companies that we have transformed the vast majority had tried at least one Agile coach/guru/system and got basically no results. In all cases it was because they failed to contextualize the techniques and model behaviors.

Use Purpose to Drive your Transformation

Another strange phenomenon we see is that when people implement Agile they seem to ignore the same playbook that works for virtually any other project: define your objectives, know what you’re trying to fix. Much of the Feckless Agile out there comes from the project being defined as: teach people the Agile techniques. You now know that is both mostly needless and somewhat futile.

Instead, contextualize your Agile around things that it can help you do better. The managerial failures list is a good starting point — which of those failures hold back your teams the most, or impair you getting a good work product out the door, or delivering on time? When you implement a technique, don’t think about doing the technique “right” as there is no right way, but rather, think about how to do the technique in a way that satisfies the objectives you’re trying to accomplish. Contextualize it.

Jack is a former RAND senior analyst, recovering client services executive, and the founder of the Agile transformation and coaching firm that helps agencies, marketers, and other complex service organizations, AgencyAgile.