Do current models of commissioning evaluation support good evaluation practice and meet their intent?

How evaluations are commissioned affects a wide range of people involved with both the activity being evaluated and the process of the evaluation. The funder has a lot at stake to ascertain the benefit of the activity[1] and the cost of the evaluation. So too, do other parties and participants who value the activity and/or contribute their time, knowledge and experience to the evaluation.

This is the first of three blogs emerging from a workshop about commissioning organised by the AES Fellows at the 2025 AES conference. It complements other recent presentations about alternatives to traditional methods of procurement of evaluations.

[1] The generic term 'activity' is used to incorporate: program, service, strategy, event, facility, system

BLOG 1: SCOPE Who does a commissioned evaluation involve and what matters to them?
BLOG 2: OUTCOMES What commissioning processes facilitate good evaluation practices?
BLOG 3: FUTUREWhat would it take to shift systems and culture to facilitate good evaluation outcomes?

The blogs report on contributions from 80 workshop participants. Now we want to expand the conversation to add YOUR experiences. After each section, there are questions for you to respond to. These are collated at the end. We look forward to your additions and comments.

Why is the role of commissioning an issue for evaluators?

Evaluators are set high ethical standards for conducting evaluations based on core values and principles (AES Code of Ethics). AES members 'have a responsibility to all stakeholders (including participants / target audiences) involved in evaluation, as well as to the broader public interest'. While there is no formal qualification or registration of the 'profession' of evaluators, the AES has developed the Evaluators' Professional Learning Competency Framework as a resource to support professional practice and development.

Hence the context of an evaluation needs to enable evaluators to meet their professional responsibilities, and achieve to the extent possible,'quality evaluations that make a difference' (Australian Evaluation Society goal).

The increasing complexity of public services challenge the traditional transactional models of commissioning. With greater acknowledgement of both the obligation and benefit of closely involving service users and communities throughout the evaluation, collaborative and relational commissioning approaches are being tested.[2] However, widespread take-up is hindered by potential increase in costs, time, risks and evaluative skills.

The first step is to identify if, where, and to what extent, issues exist across the commissioning of evaluations. Scoping the current commissioning environment will provide a basis on which to explore how different parties are impacted and what aspects of evaluation are of value to them.

[2] For example:systems-focused, relationship and partnership models of commissioning evaluations (e.g. See Human Learning Systems e.g. Lowe T. Plimmer D. Exploring the New World: Practical insights for funding, commissioning and managing complexity. Collaborate Collective, University of Newcastle and video discussion

Who does a commissioned evaluation involve?

An evaluation involves many different parties with different purposes, responsibilities, roles and values. There are numerous policies, standards, protocols and regulations that frame aspects of the evaluation and its commissioning and many of these are unstated.

Consider: Who do procurement decisions impact on?

Who has a direct stake in the evaluation: what it covers, how it does it, what it looks for and the impact it makes?

Who else is impacted indirectly?

Who bears the cost of the evaluation being undertaken (including non-monetary costs)?

Who owns the evaluation?

Box1 and Box 2 below list parties identified by the AES Conference workshop participants.

Box 1 considers parties who are involved in the purchase and delivery of the activity (program, service, strategy, event, facility, system) being evaluated.

Box 2 considers the parties involved in the purchase and delivery of the evaluation.

What do the different parties and individuals want from the evaluation process and the evaluation outcome?

Workshop attendees identified many attributes of the procurement of evaluations that are likely to be important to each of the participants above. Some attributes were attributed to multiple parties, so below they are grouped around:

a. the procurement process itself

b.the evaluation process that the procurement documents authorise and specify

c.the evaluation outcomes that are required by the procurement.

a. The procurement process itself

The procurement process is caught between:

-regulatory issues requiring fairness, probity and secrecy in the selection and use of public money

-expectations and benefits of grounded, inclusive decision-making that require transparency and collaboration.

The choices made in the procurement design may have direct impact on factors such as who can undertake the evaluation, what is valued in both assessing the tenders and in what counts as evidence, who can contribute, what aspects of the program matter.

Box 3 shows attributes of the procurement process that workshop participants identified as being important to some parties.

b. The evaluation process which the procurement documents specify

The way in which the evaluation is conducted depends on the agreed contract emerging from procurement process, how it is monitored and variations that are negotiated. The relationship between the funder and the evaluator may range from being close partners to being completely at arms length. The funder may be a critical source of access to participants, experts and other informants, relevant documents and data, background information and future directions.

Box 4 shows attributes of the evaluation process that workshop participants concerned are important to some parties.

c. the evaluation outcomes that are required by the procurement

The outcomes specified in the tender and contract set boundaries on the evaluation methodology including quantifying and describing, for example, 'what happened, how much/ many, who by, who for, its worth, what 'worked', what didn't work and why - in line with the premises of the theory of change.

Credible outcomes can only be achieved if adequate, reliable data representative of the cohort, is available within the timeframe, is well understood, and has a convincing link between the activity undertaken and the identified change.

Box 5 shows attributes of evaluation outcomes specified by procurement documents that workshop participants considered are important to some parties.

Achieving quality evaluations that make a difference

The commissioning of evaluations has to take into consideration that evaluation is considered to be applied research and as such must meet ethical guidelines and legislation (e.g. privacy and information-sharing). Depending on the design, evaluations may need to meet rigorous requirements of human research:

The Australian Evaluation Society Evaluators' Professional Learning Competency Framework notes that an evaluator is surrounded by, and works within, a multiplicity of value perspectives, including cultural, social and political... The evaluator must be cognisant of, and responsive to, such value perspectives'.

Guidelines and Frameworks have been developed to guide those responsible for evaluating activities involving Aboriginal and Torres Strait Islander peoples to ensure that cultural expertise is embedded.

Similarly, how does Commissioning take into account the inclusion of people with lived experience?

PLEASE TELL US YOUR EXPERIENCE

Q1. Who else is involved in the purchase and delivery of the activity being evaluated?

Q.2 Who else is involved in or impacted by the purchase and delivery of an evaluation?

Q3. What other attributes of the procurement process are important?

Q4. What other attributes of the evaluation process are important?

Q5. What other attributes of the evaluation outcomesare important?

Q6. What other guidelines and quality standards are central to quality evaluation practices?

This post was written by AES Fellows Marion Norton, John Guenther, Patricia Rogers, Lyn Alderman and Alan Woodward.