From the AES President

Evaluation: Adding Value

The 2008 Perth conference theme, Evaluation: Adding Value, gave us all an opportunity to look at the ways in which evaluation enhances, or not, the value of programmes, policies and practice as well as the opportunity to perhaps question some of our taken for granted assumptions around evaluation. The foci was on asking ‘value for whom’; ‘whose values’; and ‘optimising value’ - areas addressed by keynotes and other speakers over the conference.

Why should we be looking at this? If we consider ourselves to be part of a ‘community of practice’ – defined as groups of people who share a concern, set of problems or passion about a topic – then evaluators meet this criteria. We may share concerns around an issue or topic, or it may be that evaluation itself is our concern or passion. We may have taken a variety of routes to the evaluation community of practice and each of these different routes will have influenced the way in which we value evaluation and the approach we take to it. In the Australasian context, Australian universities have a long history of offering qualifications in evaluation while in New Zealand this is still a developing area with greater reliance on professional learning opportunities. Therefore, it is particularly important that our annual conference address areas that require us to think again about what it is we do and why.

There is a ‘taken for grantedness’ about the value of evaluation as a ‘good’ thing that helps informed decision making. This is clearly the underlying rationale behind the enthusiastic adoption of evaluation by governments and possibly reflected in the evidence based movement which started in the health sector. NGOs and other community groups have to ‘prove’ that the projects they are funded for ‘work’ and deliver the outputs required of them as does the government sector itself, particularly in times of scarce resources, competing imperatives and politicians’ pet projects. We are all very aware of the political context within which evaluation takes place and what needs to be balanced at all stages of a project.

This brings me to the first of the interlinking themes – value to whom. In any evaluation there are at least three parties/participants: the commissioner of the evaluation; the evaluator; and the stakeholders. All three of these groups are looking for ‘value’. The commissioners want value – value for money, to see whether the programme, policy or practice is working well, is delivering value for them. Stakeholders also want value, but focused on their concerns. As evaluators, what do we want? To start with we want our evaluation to be valued. The utilization focus espoused by Michael Quinn Paton resonates with many evaluators rather than an approach where completing an evaluation is seen as a tick box exercise, something that has to be completed to meet compliance requirements - as an end to itself.

However, even in this latter situation evaluation can have value for the evaluator in terms of lessons learnt, better knowledge of the political realities of the situation and perhaps the community where the evaluation is taking place. There are times when we may feel somewhat disillusioned about the value of what has been done subsequent to an evaluation. Reports are ignored, people take issue with what was concluded or recommended. However, it is worth remembering that although there may appear to be no direct link between the evaluation and subsequent action, or lack of it, the evaluation provides input to the wider context within which the programme, policy or practice is discussed and considered. Rather than a direct line between the evaluation and subsequent action, and recognition of its value, the evaluation permeates the terrain and findings and discussions around them inform future decisions.

This then moves us to ask whose values? Dr Fiona Cram, in her keynote at the Perth conference, discussed the principles for evaluation underlined by the value of respect – respect for all those involved, and the way in which evaluation can be located outside the community where the evaluators helicopter in and take the information away which negates this principle. Power equals whose values count. She also discussed the need to develop policy frameworks where values are made explicit and further that shared explicit values help the process of evaluation for all those involved.

Being aware of whose values are being espoused feeds into recommendations around policy, programmes and practices and at times their acceptability and ‘actionability’. Again, there is a very fine balancing act involved. At the 2005 Brisbane conference, David Turner and others presented a paper comparing AEA and AES members’ approaches to ethics and standards in evaluation and one of the major areas of difference was the importance that AES members placed on responsibility to community. Therefore, the question of whose values may prove to be more of a challenge as we consider how to make decisions on this aspect. It moves us into that ethical dimension, from the benign ‘do no harm’ to the active ‘leaving people better off’ as a result of the evaluation – but how?

This leads us to the final area, that of optimising of value. As Gaye Mackenzie, convener of the Perth conference, said in the call for papers undertaking evaluation inevitably involves trade-offs – in the way the evaluation is conceptualized, carried out, results presented and the utilization of the findings. Keynotes at the conference discussed involvement versus the rush to implement, and the need to do more than just talk to the stakeholders. We are all aware of the debates around methods used and the favoured theoretical positions of commissioners, evaluators and other users of evaluation when the ‘fit for purpose’ imperative is ignored or lost. Given time constraints and political constraints how do we optimize value?

To start with, we need to be aware of what these constraints are, to see how they can be worked with, worked around to work for the best interests wherever possible. The importance of clarity around the purpose of the evaluation should not be lost as that is crucial to ensure all that can be done to optimize value is done. Finally, in thinking about adding value, once completed - understanding how the evaluation could have been done better is the ultimate step. The perfect evaluation has yet to be done.

I would like to wish you all the best for the festive season

Associate Professor Jenny Neale - AES President

AES 2009 Canberra Conference

Evidence and Evaluation

September 2009

Over the coming months the Canberra Region, with Conference Chair Graham Smith at the helm, will be gearing up for the 2009 AES Conference. Featuring exciting speakers and an emphasis on interactive formats the conference will explore the centrality of evidence to evaluation. Themes and subthemes for the conference include:

Gathering Evidence

• collecting evidence for evaluations
• analysis of evidence in evaluation

Using Evidence

• evidence-based policy
• evidence to support change
• using evidence for program development and implementation

Evidence and Stakeholders

• engaging stakeholders in order to gather sound evidence
• giving stakeholders evidence for the effectiveness of evaluation

All members will receive a copy of the registration brochure.  Keep an eye on the AES website for updates at http://www.aes.asn.au/conferences/ . Canberra region members who would like to be involved in organising the conference can contact Graham at: GrahamSmith@bigblue.net.au .

AES 2008 Conference Papers and Photos

Members are reminded that the AES website currently has PowerPoint files, papers and a wide range of photos from the AES Perth conference. These can be accessed at: www.aes.asn.au/conferences/ .

Professional Learning Committee

The Professional Learning Committee presented a symposium and a poster at the recent AES Conference in Perth , with a view to gathering information from delegates on professional learning issues and needs in regards to evaluation.  This proved to be very helpful, confirming information previously gained from discussions with regional groups as well as raising some additional issues and possibilities.  Sincere thanks are expressed to everyone who participated in these data gathering activities.

The Committee is now preparing a statement on needs that will be tested in a number of focus group discussions early in the New Year before being presented to the AES Board. In the meantime, a series of national workshops is being planned for April-May 2009, to complement other activities organised by the regional groups.  Details of forthcoming events will be publicised on the AES website as they become available.

Dr Paul Chesterton - Committee Chair

Research Methods Training

ACSPRI

Date: January, 2009

Where: The Australian National University , Canberra

Need to brush up on qualitative or quantitative methods? Consider attending the ACSPRI summer program. According to the ACSPRI website:

“The ACSPRI Summer Program comprises two sets of week-long courses in social research methods and research technology. This program is designed to cater for fundamental, changing and emerging research strategies, and to serve a wide variety of needs for training and professional development within the academic, public and private sectors.

Learn more about summer program at: http://www.acspri.org.au

Call for Presenters

In 2009 the AES is inviting members travelling within Australia and New Zealand to consider providing a lunchtime/evening seminar or even a workshop (on a topic of your choice) to members of a region that you are travelling though. Please contact the AES Administrator for further details: aes@aes.asn.au

Get Involved in Your Regional Branch

To contact your region representative or to access the latest reports and news on upcoming activities at your local AES branch click on the 'Regions' link on the AES website. www.aes.asn.au/regions/

Welcome New Members

Auckland

Canberra

New South Wales

Northern Territory

Queensland

South Australia

Tasmania

Victoria

Western Australia

Wellington

International

Contact the E-News Editor

If you have contributions or a notice for E-news please email the E-news editor, Brad Shrimpton bshrimpt@unimelb.edu.au