The shortcomings of agile coaching – challenges and solutions

Recently (on July 20, 2023), I had the honor of participating in an event as part of the Agile Munich Meetup. The format of the event was an “Oxford Debate”, a highly structured debate in which proponents and opponents are asked to do their best to convince the audience. The moderator conducts polls of the audience both before and after the debate on their position on the key question.

The debate’s leading question was: Are agile coaches catalysts for teams and organizational progress? I was part of the opposing team and had to convince the audience that coaches are not catalysts – an interesting challenge with my background as an agile trainer and coach for many years.

The very stimulating and interesting debate led to this article: I try to reformulate my arguments – a bit more balanced – and add some consequences. As well as pointing out shortcomings, I want to contribute to a longer-term debate: if agile is heading in the direction of a commodity, what is the next big thing? Should we really be thinking beyond Agile and what theories/methods/frameworks/none-of-the-above come into question?

Introduction

Agile coaching has gained significant traction in the field of software development and project management due to its ability to foster adaptability, collaboration and continuous improvement.

Agile coaches help teams and organizations adopt agile methodologies. However, despite its many benefits, agile coaching is not without its shortcomings. In this article, we look at some of the most common challenges faced by agile coaches and discuss possible solutions to effectively address these shortcomings.

The challenges

• Lack of clarity in roles and responsibilities. One of the main shortcomings of agile coaching is the lack of clear definition and understanding of the role itself. Agile coaching can be ambiguous, and organizations may have different expectations of the coach’s responsibilities. This often leads to confusion within the team (and the organization as a whole) as some team members do not know what authority the coach has or how they fit into the existing hierarchy.
• Insufficient experience and expertise. Agile coaching requires a deep understanding of agile principles, methods and best practices. However, knowledge of agility is not enough: Coaches must also have the expertise to deal with the existing structures, reward system and all the other realities in an existing organization. Unfortunately, there is a shortage of experienced and qualified agile coaches in the industry. This is exacerbated by the fact that the expectations of the coach’s role are so unclear. This can lead to organizations looking for a pink unicorn and end up working with coaches who don’t have the expertise to effectively tackle the specific, complex challenges.
• Difficult to measure impact Measuring the success of agile coaching can be challenging. Unlike other roles in a project or the roles of managers, the impact of coaching is often intangible and difficult to quantify. Coaches in an organization-wide role also run the risk of taking on a proxy role for a manager, making the roles and their responsibilities even more unclear. Managers may feel challenged in their role and organizations may therefore struggle to demonstrate the return on investment (ROI) for hiring agile coaches, leading to skepticism about their effectiveness.
• Resistance to change. Agile transformations (like other change initiatives) often encounter resistance at various levels of an organization. Team members may be reluctant to adopt new methods and management may be reluctant to relinquish control. Agile coaches then face the challenge of overcoming this resistance while fostering a culture of openness and collaboration. If the coaches are hired to overcome resistance without changing the system as a whole, they are doomed to fail – and they often leave a lasting negative memory of the initiative or of Agile as a whole.
• Building a shadow structure in the organization. Agile coaches need to justify their role and secure their position like everyone else. For external coaches in particular, this means a preference for long-term contracts and embedding themselves in organizational structures. This mirrors the development of the traditional consulting industry, where large consulting firms are deepening dependencies and amassing more and more expertise away from their clients. It makes accountabilities less clear, it can become toxic when coaches are used by managers to be “off the hook” when it comes to driving change, and it is counterproductive: organizations tend to react like a metaphorical rubber band that returns to its starting position when the pressure is released.

Doing things better and doing things better

It is more than a truism that you have to both do the right thing and do the right thing better. I’ll try to give a few nudges in both directions.

An initial set of ideas for improving things – in coaching and beyond

None of these ideas are really new, but many people still fall into a stubborn “business as usual” instead of changing the way they do things. I will try to collect some of these ideas here:

• Making responsibility transparent. Edgar Schein distinguishes between “expert consulting” and “process consulting”. While an expert consultant has expertise in, let’s say, diesel engines, a process consultant helps the organization to perform. In doing so, the process consultant develops a “helping relationship” with the client. Does that sound familiar? Edgar Schein wrote about this in the 1980s [1]. If we add Kühl’s “person-oriented consulting” [2], which includes mentoring, coaching and similar activities, a much clearer spectrum of roles emerges. If you can’t come up with Chuck Norris or a pink unicorn, you might be better off having a clear notion of the job to be done and hiring on that basis.
• Train and empower. Don’t just change role names. Scrum Masters receive a certificate in a two-day course. This is a license to learn how to be useful to a team if you have some experience in teamwork and some talent. You need a learning environment and can benefit from a mentor. When a company provides the opportunity to learn, it represents a significant investment and a shift to a learning organization. This will pay off in many ways: more motivation, more flexible employees, more effectiveness, more adaptability and viability. On the other hand, if a Scrum Master becomes a coach by attending another 5-day course and manages not to die in that time, this becomes ridiculous. We need a more serious approach to consulting/coaching.
• Don’t try to change a system against its will. Sustainable change and improvement must be anchored in the structures and culture of the organization. If it is easier to fall back into old patterns, people will fall back[4]. No amount of thinking or exhortation will change that. This means that an intervention should be an experiment that can improve the conditions for subsequent initiatives, such as micro nudges[3] (found thanks to Dave Snowden) that slowly turn the tide.
• Change the system, don’t rely on crutches. Consultants (this includes coaches) can be very helpful as temporary scaffolding, but they can also be harmful if you let them become permanent crutches. Don’t let managers off the hook. Let them do their job and don’t allow them to rely on crutches. Your job is to create an environment where everyone can be effective, creative and thrive. This cannot be delegated to a coach or a consultant, but must be an integral function of the system. Managers must be enabled, empowered, motivated and rewarded to accomplish these tasks.
• Looking beyond agile. Well, this already falls into the category of “doing the right thing”, so let’s move on to the next section.

A few directions that go beyond Agile

A big shift is underway in Agile. The shift towards commoditization (one of my clients translates that as “bulk”). This means standardization, but it also means that there is a relatively stable scope and state of the art. Some people try to extend the meaning of agile to other areas – I’m not a fan of that. It might be better to leave agile where it makes a difference and look for greener pastures.

This is the result of my research over the last few months (some going back years). It is a personal selection of tools that I have found useful, plausible or enlightening. It is by no means complete, but – for me at least – a starter kit for the next big thing.

• Systems theory – more than systems thinking. Systems thinking, i.e. thinking and working with feedback loops and complex interactions, is a powerful tool in itself. It can help to question and break through the control illusion of the die-hard machine logicians. But there is more. First, there is a whole family of systems theories, each with a rich practice and many with a solid theoretical foundation. They range from cybernetics as the science of control, the origin of all these theories, to human systems dynamics, soft systems theory and many others. One of the most important for me is the Viable System Model, which refers to decision-making and communication processes in organizations. You will find a myriad of practices that are also used in agile contexts. The great strength of these books and papers is that the authors put a lot of emphasis on the empirical verification of their work and cite their sources extensively – which is not the case with all authors in the agile field.
• Complexity tools – thechampion sceptic among theories. While cybernetics – and the theories derived from it – claim to be the science and theory of control, complexity theory says: there is no such thing as control in a complex adaptive system. This is an interesting challenge. It prevents systems theorists from selling their cause as a panacea and brings it down to the level of tools. Tools can be combined; their effectiveness depends on the people who use them. Furthermore, there are some unconventional ideas and approaches in the field of complexity theory that can be used to break down entrenched structures and old thinking and gain new insights.
• Back to the roots of Agile: Lean. Lean is one of the main sources of Agile. If you look at the early Scrum papers by Jeff Sutherland, you could say that if you apply Lean to the software industry, you end up with Agile – well, at least with Scrum. An example of going back to basics would be: teach structured problem solving. Structured problem solving is tailored to complicated environments, not complex ones, but firstly, in every complex situation there are complicated parts, and secondly, and more importantly, it’s a great example of how to plan your work thoughtfully and execute it well. And that, my dear monastic brothers, is not a matter of course everywhere.
• Organizational design and organizational development. If you play with the acronym OD, you could add: Organization Debugging. Many of the people in this field are firmly rooted in traditional change processes, but not all of them. OD has a big advantage over Agile: OD tries to get a complete overview of the different dimensions of an organization, while Agile and Lean focus heavily on processes. Again, while Agile has provided value, it’s time for a more general approach.
• Work on the basis of how people behave, not how you want them to behave. Working on the way people think is intrusive and – this is my argument – it doesn’t work. If we want to change people’s behavior in an organization, we have to change the conditions. This includes the opportunities and resources available to them, the degree of freedom to make a real difference with their actions, and the way relationships work in the workplace. It also includes trust and personal security: they are real and not part of the mindset.

[1] Schein, Edgar H. 1999. Process Consultation Revisited: Building the Helping Relationship. Reading, Mass: Addison-Wesley.

[2] Kühl, Stefan. 2008. Coaching and Supervision. Wiesbaden: VS Verlag für Sozialwissenschaften. https://doi.org/10.1007/978-3-531-91136-6.

[3] “Nudge theory – Wikipedia”. n.d. Accessed July 28, 2023. https://en.wikipedia.org/wiki/Nudge_theory.

[4] “Assemblages, Agency and Affordances- a discussion with Dave Snowden and Ellie Snowden”. n.d. Accessed July 28, 2023. https://cognitive-edge.wistia.com/medias/otsev5jq9e.