by Jade Maloney
Ever found yourself more engaged in the coffee break than the conference agenda? Ahead of the International Evaluation Conference #aes19SYD unconference day, Ruth McCausland, Kath Vaughan-Davies and I trialled an approach for the Australian Evaluation Society NSW meet-up that combined the best of both worlds – purposeful encounters with a coffee break vibe.
We adapted Open Space Technology, established by Harrison Owen in the 1980s, with the aim of finding “a way towards meetings that have the energy of a good coffee break combined with the substance of a carefully prepared agenda.” The approach has since been used around the world as a way of enabling people to self-organise around purpose.
At the NSW meet-up, about 30 evaluators braved the wind and cold to talk about evaluation topics that keep them up at night. For those of you in evaluation, it will be no surprise that these were many and varied:
- managing your involvement in participatory action research
- communicating findings effectively, particularly the negative
- scoping evaluations effectively to meet and manage expectations
- identifying value and dealing with attribution in an education context
- planning for the data required for statistical analyses and the ethics of analysis
- crafting useful and useable recommendations.
And that was before we got to our back-pocket topics
Working what we dubbed the “East Wing”, the “West Wing” and “next door”, groups took their discussions in different directions.
The group discussing evaluation in an education context shared references: the four levels in Kirkpatrick’s Evaluating Training Programs (reaction, learning, behaviour and results), Guskey’s additional fifth level (although organisational support isn’t a level in the same way), as well as Mitchie, Stralen and West’s COM-B system (thinking about behaviour change in terms of capability, opportunity and motivation).
The participatory action research group was prompted by a question from one evaluator about whether he'd become too involved. They segued into how an evaluator’s participation can shape what is being evaluated and questioned whether this matters. The many lines between questions and the “really??” underneath the word "objective" in their record capture the connecting threads of their conversation, but you had to be there for the depth.
Instead of a traditional report back, we came together as we began – in a circle. The energy was palpably different, shifting from hesitant suggestions to each person sharing something they’d take forward and participants building on each other’s thoughts.
Some focused on practical tips, such as taking the time to clearly scope evaluations upfront and having findings meetings before delivering reports; some on tools (like the COM-B system); others on the process. One participant described it as bringing to life a community of practice in the AES. A few said the problem they’d started with might still keep them up at night, but they felt less alone in it. While we all came from diverse backgrounds, we found common ground among our experiences in NGO, government and private sector evaluations.
Not having set questions to answer gave people the freedom to discuss what they wanted and to go deep on a subject, and the process enabled all to have a voice.
Want to experience the process for yourself? Come along to the #aes19SYD unconference on Tuesday, September 17, to discuss how we might un-box evaluation to better contribute to reconciliation, social justice and a healthy planet. You don’t have to have the answers – just a question and the passion to hold a discussion with others on the subject.
If you’d like to learn more about Open Space Technology, there are a wealth of resources online. Chris Corrigan’s website has an easy-to-navigate collation. Or you could go back to the source: Harrison Owen’s Open Space Technology.
Jade is a Partner & Managing Director at ARTD Consultants.