Evaluation – more for less?

Over the past few months, I’ve been working with some actuaries: highly trained people who specialise in analysing numbers, statistics, risks and outcomes. We’ve been part of the Actuarial Hackathon where 40 young actuaries volunteered their time to collaborate with five Australian charities – the Wilderness Society, Justice Connect, Social Ventures Australia, UNICEF Australia and the Community Council for Australia – to transform complex datasets into practical solutions that are already driving real-world change. I’m proud to say the actuaries CCA worked with won the award for the best pro-bono actuaries’ program presentation.

I can’t yet talk about the full nature of their work with CCA, but it involved analysing the last 10 years of financial data for all charities on the ACNC register. Through trial and error, the group developed a “financial strength scorecard” for charities which they mapped across all charities to prove the scorecard they developed was predictive of whether a charity was more or less likely to fail in the next two years. In our early discussions, I was sceptical that financial information alone would allow predictions about future sustainability, but the performance of their financial strength scorecard surprised me.

In reviewing the findings, we talked as a group about how important outcome measures are in assessing the work of charities, and the limitations of focusing solely on financial indicators. This led to more discussions and debate about randomised controlled trials (RCTs), which the actuaries saw as the gold standard for measuring whether charities were delivering on their stated objectives.

These actuaries are not alone in this view. The Assistant Minister for Charities, Dr Andrew Leigh, has also championed RCTs. A recent review of the use of RCTs in the USA by researchers from the University of Chicago (Marwell and Mosley) found that both the use of RCTs by not-for-profits, and discussion about RCT findings about not-for-profits, has been significantly increasing over the last 20 years.

While RCTs often provide the clearest insights into what works and what doesn’t, especially in areas like medicine, I’m not so sure they should be seen as the best form of research when evaluating organisational effectiveness or the programmatic performance of charities and NFPs.

In their book Mismeasuring Impact: How Randomized Controlled Trials Threaten the Nonprofit Sector, Marwell and Mosley argue there are five main reasons RCTs are not always the best evaluation instrument.

My own experience is that there are any number of issues with implementing RCTs within charities and NFPs, not the least of which is the extraordinary amount of time and effort it takes to ensure appropriate levels of randomisation, standardisation of the intervention, and monitoring of critical indicators (often over years), as well as the enormous cost in staff time (countless hours of additional work), and the inability to capture and control for important variables such as affect (how people feel about each other), or broader community engagements and impact.

The bottom line is that there’s nothing standardised about how people provide and receive support, care and other interventions.

Of course, the difficulties with RCTs shouldn’t stop the charities and NFP sector focusing on outcomes and measuring impact. We’ve known for some time that our sector needs to get much better at demonstrating our value.

But perhaps the biggest weakness we have in evaluation and demonstrating value is not a lack of RCTs, it’s a lack of access to connected-up data.

Imagine if we could link health, education, socio-economic, geographic, justice system, social services, housing, employment and financial information together and use that information to monitor how engagement in various services affects outcomes?

What if we could link together the information charities and NFPs provide to governments and others about clients to the government’s own information about the services or payments provided to people? What if all government grants to not-for-profits were on a database with a shared dictionary, and this could be linked to government and sector data?

There is increasing pressure on almost every charity and NFP that receives grants to evaluate more, or better. The focus of this pressure is often the creation of more or new information. Is that really what is needed, or do we need to better use what we have?

I have watched as incredibly smart people have written analytical algorithms and applied them to a large data set provided by our sector (through our annual information statements) – data we have access to because government, when it established the ACNC, agreed to make the data collected from the sector available back to the sector. It made me realise yet again how good it is to have access to that data, but also how far behind our sector is at leveraging what we already know, the information we already have, the data we already provide.

With productivity driving discussions at a national level, now is a good time for charities and not-for-profits to highlight how government departments and other funders seem to be demanding more and more information, data and evaluations from our sector, but the same funders rarely provide any data or evaluations back to the sector or the community.

Why do we continue to support the lack of data symmetry in the sector’s relationship with government and other funders? Maybe we start a “You show me your data, and I’ll show you mine!” campaign? At the very least, we need to point out that if we had better, more accessible, connected government data, the need for every individual charity and not-for-profit to provide more and more evaluation data would be greatly reduced.

David Crosbie has been CEO of the Community Council for Australia for the past decade and has spent more than a quarter of a century leading significant not-for-profit organisations, including the Mental Health Council of Australia, the Alcohol and Other Drugs Council of Australia, and Odyssey House Victoria.