Insights from a study by Lyn Pleger and Franz Leeuw

I came across the edited book Ethics for Evaluation, by leading international authors Rob van den Berg, Penny Hawkins and Nicolette Stame (2021) when preparing a workshop proposal with AES Fellow John Stoney, for an ANZEA conference in 2024. We wanted to get people talking about being resilient in the face of the political cycles that are the complex backdrop to our work.

While this book is about ethics, it's also about something else really important: what supports resilience.

Buried within this edited volume is chapter 7, 'Resilient Evaluators: Characteristics, Conditions and Prospects', which highlights the importance of both individual and institutional components of resilience. The chapter authors, Lyn Pleger and Franz Leeuw, are a rare kind of evaluation specialist – those who support the field by doing research about the practice of evaluation. The chapter draws on their findings from a small study involving interviews with 21 practicing evaluators.

It's not always easy to get a hold of a book or find time to read a chapter…but in these challenging times, the topic of resilience seems highly relevant. So I'm sharing my reflections on a key insight that I gained from this chapter.

The individual and institutional faces of resilience

What I find especially interesting about the coverage of evaluator resilience in this study is what it reveals about both individual and institutional dimensions.

Pleger and Leeuw provide a model of Evaluator Resilience with 8 dimensions: character traits, methodological skills, integrity and soft skills relate to individual dimensions of evaluator resilience, whereas evaluation management, code of ethics, training/professionalism and work environment and structural independence are about institutional arrangements.

A few of these dimensions seem especially relevant to considering adaptation to cycles of complex systems.

All 4 of the individual competencies are important, but 2 in particular may boost evaluator resilience when responding to political cycles:

• Soft skills – communication, flexibility and adaptability
• Integrity – independence of thought, an "ethical backbone" and professional self-confidence.

Perhaps less familiar than individual competencies are how the institutional features of organisations play a range of roles in supporting structural independence:

• Codes of ethics can guide the conduct of evaluation, but Pleger and Leeuw also found that these can act as references or anchor points that can be referred to in discussions with clients, as they may lend authority to a position.
• A 'resilient fostering' work environment is a rich concept, and this is summarised as "an institutional arrangement that puts a high value on credible and valuable evaluations and protects independence in this light" (p 152). This draws on a set of features that structurally foster resilience. Examples provided cover organisational clarity about commitment to independence, contextual support for evaluation conduct, mechanisms that stimulate debate with stakeholders, and recruitment procedures that provide competent staff.

Implications

My reflection on these particular findings of this study is that we need a system that structurally supports debate and rigour, via frames of reference, processes, training and mechanisms that foster a resilient system, recognising that there is value in both internal and open discussion. Evaluation leadership is sometimes seen as an important dimension, but this chapter highlights organisational features which can endure if the enthusiasm of leaders' wanes. It also highlights the importance of organisational systems that value evaluation skills and competencies and, in addition, seeks ways to boost the spread of trained people across the organisational roles involved in planning, commissioning, doing and using evaluation. Broadening the spread of skills across different parts of the system could help foster shared responsibility for quality evaluation.
I'd welcome your thoughts.

References

Pleger, L.E, & Leeuw, F.L. Resilient Evaluators: Characteristics, Conditions and Prospects, in van den Berg, R.D., Hawkins, P., & Stame, N. (Eds.). (2021). Ethics for Evaluation: Beyond "doing no harm" to "tackling bad" and "doing good" (1st ed.). Routledge. https://doi.org/10.4324/9781003247234

Kim Grey

Kim has experience in evaluation spanning 3 decades. As an experienced manager of evaluation across many areas of social policy, Kim has insight into strategic organisational governance of agency evaluation work plans, and capability development approaches for the range of staff who touch on the journey from planning to doing and using evaluation. Kim has worked inside government as an internal evaluator and led teams focused on evaluation capability development and evaluation governance. Kim has also conducted and provided training in evaluation through academia and consulting.

Kim follows the broad field of evaluation, but has particular interests in impact evaluation and participatory and realist evaluation. She is passionate about the evaluation profession and deeply curious about how we build resilience.

Kim has a Master's Degree in Evaluation from University of Melbourne, and has completed research about the use of theory in evaluation.

Kim is the current President of the AES. Her commitment to the AES includes participation in the ACT regional committee, the Policy Committee and the Editorial Board of the Evaluation Journal of Australasia, as well as being on the organising committee for the Canberra conference in 2017 and co-convenor in 2025.