Workshop: Using Systems Thinking and Boundary Analysis Strategies to Advance Culturally Responsive Evaluation Planning and Design

Date and time: Thursday 14 November and 21 November, 10.00 am to 1.00pm AEDT (registration from 9.45am) Registrants are to attend both sessions. (full day workshop - 2 sessions)

Venue: Via Zoom. Details will be emailed to registrants just prior to the workshop start time

Facilitators: Dr Jori N. Hall, Ms Eva Sarr and Mr. Bob Williams

Register online by: 13 November 2024, unless sold out prior limited to 25 participants.

Fees (GST inclusive): Members $295, Organisational member staff $415, Non-members $485, Student member $140, Student non-member $226. Students must send proof of their full-time student status to This email address is being protected from spambots. You need JavaScript enabled to view it. 

Please note: Participants will require materials to bring to the sessions including but not limited to various coloured post it notes; white board markers; paper and more. Details will be included closer to the date. 

Workshop Overview

The significance of Culturally Responsive Evaluation (CRE) and its compatibility with systemic evaluation approaches like “Systemic Evaluation for Gender Equality, Environments, and Marginalized Voices” (ISE4GEMs) is increasingly being recognised in the evaluation field. ISE4GEMs offers a systemic lens, complementing CRE's focus on “cultural contexts” in all phases of an evaluation to address issues of equity, inclusion and power.
Integrating ISE4GEMs tools into CRE can enhance the “how” of culturally responsive and equity-focused evaluation practice and in shaping the design of interventions aiming to address social justice and inequity.
This workshop's emphasis on the CRE preparation phase, along with the incorporation of two ISE4GEMs tools which will be adapted for this workshop: the boundary analysis to delineate what is ‘in’ and ‘out’ of both the intervention and the evaluation, and a specific form of stakeholder analysis that examines their roles, power, marginalisation and impacts.
Stakeholder analyses reveal the different perspectives and thus motivations and commitments to collaborate with or oppose the design and evaluation of an intervention. Boundary analysis and investigations of boundary choices highlighting the values that underpin key decisions in what to include and exclude, ignore or marginalise in an intervention’s design or its evaluation. Evaluation criteria are essentially boundary choices because they delineate what is worthwhile and what is not.
Designed for mid-career and seasoned evaluators, the workshop offers valuable insights into innovative evaluation practices, aligning with current needs and leveraging facilitators' expertise.

Workshop Content

The facilitators will use the following online adult teaching/learning principles and approaches that allow participants to practice ISE4GEMS boundary analysis tools with CRE principles.

Following is a list of adult learning principles that we will incorporate into the workshop activities:

• Goal-oriented approach: Workshop sessions have specific objectives, with participants encouraged to define their own learning goals beforehand, fostering accountability and relevance to individual needs.

• Self-directed and small group learning: Visual aids and handouts will be utilised to introduce the CRE framework and ISE4GEM approach, focusing on boundary analysis and stakeholder tools. This will provide foundational knowledge for active participation.

• Collaborative large group work: Using a case study, participants will explore application of ISE4GEM tools to CRE planning, particularly focusing on equity issues, social justice and cultural intersections, preparing them for real-world evaluation projects.

• Practical and problem-centered: Discussions will focus on applying ISE4GEM tools to participants' own evaluation projects, addressing cultural intersections and inequities, with feedback provided for improvement.

• Respect for prior learning: The workshop acknowledges participants' existing skills and aims to build upon them in CRE planning and ISE4GEM tool application.

• Self-assessment and evaluation: Time will be allocated for self-assessment and reflection on the effectiveness of ISE4GEM tools in culturally responsive evaluation planning, with opportunity for participants to provide feedback on the workshop.

• Further guidance: Additional resources will be provided to support participants beyond the workshop.

Workshop Objectives

At the end of this workshop, participants will:

• Distinguish traditional evaluation from CRE and systemic evaluation approaches

• Recognize CRE stages: Prepare, Engage stakeholders, Identify purpose(s), Frame the right question, Design the evaluation, Select/adapt instrumentation, Collect data, Analyze data, Disseminate/use results.

• Appreciate the importance to equity of exploring issues of stakeholder motivation and boundary decisions.

• Apply boundary and stakeholder analyses to their evaluation practice for other culturally responsive social justice agenda.

PL competencies

This workshop aligns with competencies in the AES Evaluator’s Professional Learning Competency Framework. The identified domains are:

• Domain 1 – Evaluative attitude and professional practice

• Domain 3 – Culture, stakeholders and context 

Who should attend?

The workshop is suited to intermediate and advanced evaluators with or without prior knowledge of systemic approaches to evaluation and culturally responsive evaluation.

 Workshop start times

• VIC, NSW, ACT, TAS: 10.00am

• QLD: 9.00am

• SA: 9.30am

• NT: 8.30am

• WA: 7.00am

• New Zealand: 12.00pm

• For other time zones please go to https://www.timeanddate.com/worldclock/converter.html

About the facilitators

The workshop's purpose, focusing on supporting evaluators in implementing CRE phases, aligns with the facilitators' expertise. Collectively, we have backgrounds in various sectors, reflecting a range of evaluation experiences. We have a proven track record of conducting professional development workshops on CRE online, including presentations at AEA,  AES and the Eurpoean Evaluation Society. Additionally, the inclusion of hands-on activities ensures that participants will be equipped with knowledge and skills to apply these tools to unique evaluation contexts.

• Jori N. Hall has worked as an evaluator since 2006, and published numerous works in scholarly venues such as the Journal of Multidisciplinary Evaluation, New Directions for Evaluation, the Canadian Journal of Program Evaluation, the International Journal of Social Research Methodology, and the Research Handbook on Program Evaluation. As a result, she was asked to serve as a Co-Editor-in-Chief for the “American Journal of Evaluation”. Dr. Hall authored the book “Focus Groups: Culturally Responsive Approaches for Qualitative Inquiry and Program Evaluation” and co-edited the book, “Disrupting Program Evaluation and Mixed Methods Research for a More Just Society: The Contributions of Jennifer C. Greene”. She was selected as a Leaders of Equitable Evaluation and Diversity (LEEAD) fellow by The Annie E. Casey Foundation. Hall received the Scholarly Leader Award from the American Evaluation Association’s Multiethnic Issues in Evaluation Topical Interest Group for scholarship that has contributed to culturally responsive evaluation. She is currently co-authoring the third edition of the book, “Program Evaluation Theory and Practice: A Comprehensive Guide” and serving as a researcher for programs funded by the Spencer Foundation.

• Eva Sarr is an indigenous Serer woman from Sene-Gambia, in West Africa. She is also a 6th generation Australian woman of indigenous Celtic-Scottish and Irish-descent. Her father was Muslim while her mother, Catholic. Eva, a trained public health practitioner, applies pragmatic evaluation skills guided by prominent evaluation theories and methodologies that advocate collaboration, participation, equity, social justice and empowerment. She has held diverse roles including as a manager, evaluation lead, director, commissioner, consultant, and evaluation capacity-building specialist, in the international development, not-for-profit (NFP), Indigenous, State, and Federal Government sectors. Throughout her career, Eva has contributed to a variety of small, medium, and large-scale/complex health, education, and employment projects and programs, most of which have included various forms of evaluation capacity building. Eva has either led and/or collaborated with researchers and evaluators to design and implement mixed methods projects incorporating rigorous experimental and non-experimental designs such as field experiments, case-control analyses and pre/post assessments, population and program-level surveys, questionnaires, observational studies, cross-sectional analyses, and descriptive investigations. Eva is also the founding director of the Centre for Multicultural Policy and Program Evaluation and one of two Founding Chairs of the AES’ first Multicultural Special interest groups. In 2020, Eva was named an affiliate faculty member of the Center for Culturally Responsive Evaluation and Assessment (CREA), making her only one of two CREA faculty affiliates, based in Australia.

• Bob Williams is known internationally as working in various fields, including systems, evaluation and action research. Bob has been using systems concepts in his work for over forty years. He was originally trained as an ecologist, one of the earliest systems disciplines, and spent four years with the Systems Group at the Open University in the United Kingdom. Bob is well versed in a variety of different systems methods, although in recent years has focused on those more relevant to evaluation, such as soft and critical systems, viable systems and cultural-historical activity systems. Although he is now based in New Zealand , he has conducted organisational development and evaluation projects as well as many systems and evaluation workshops in Aotearoa, Australia, North and South America, Europe, Africa and Asia. In 2015, Bob was presented with the American Evaluation Association’s Lazarsfeld Award for his contributions to the theory and practice of evaluation – especially the introduction of systems ideas into evaluation practice".