The Smith Family's research is the base for its social impact

The latest – and sixth – Pathways, Engagement and Transitions Report, released this month, looked at students who were in Year 10 in 2020 and interviewed them again in 2023 to find out how vulnerable students were coping in the critical 18 months after Year 10. The study found that the proportion leaving school before finishing Year 12 is up from 14 per cent at the time of the previous study to 18 per cent. The percentage of those early school leavers who took up full-time work fell from 27 to 17 per cent, while those not in work or study grew from 36 to 47 per cent.

"These combined schooling and labour market shifts have already translated into poorer early outcomes for early school leavers," the report found.

The studies involved in-depth interviews with many of the participants, to capture their view of their experience of education, employment and training.

The findings sharpen a question that sits at the heart of The Smith Family's work: how does a not-for-profit organisation build the evidence base to know whether what it is doing is working, and what does it take to sustain building that evidence base over time?

Suzie Riddell, who joined The Smith Family as head of evidence and impact late last year, puts it plainly. "Ultimately, we work in these organisations because we want to see better outcomes in the world. The power of evidence is trying to get closer to understanding what works for whom, and in what context, so that we can do more of what works for the right people in the right context. And when we find things that don't work as well, that we stop doing them and make adjustments."

Riddell came to the role from Social Ventures Australia (SVA), where as chief executive she oversaw the 2024 analysis of 237 sector submissions to the Department of Social Services (DSS) on how government should better fund and work with community organisations. That process produced strong consensus: the sector needs full-cost funding, longer contracts, stronger data infrastructure, and dedicated investment in measurement and evaluation.

The Smith Family was one of those 237 submitters. Riddell is candid about how much has changed since.

"The gap at the moment is still quite big," she says. "There's a lot of ambition from those in the federal government, and in different state and territory governments, to try to better meet those needs. But the tension comes in a couple of different forms."

One of the biggest issues is addressing rising demand, without commensurate funding increases. "For most businesses, if more people want what you have, then you get more money. But in many not-for-profit service delivery organisations, that is not connected."

While demand is rising across food banks, homelessness services and crisis accommodation, and cost bases are growing through wage decisions, rent increases, and technology upgrades, government contracts can remain fixed. "That lack of indexation from government contracts to recognise that the cost of doing what we do has gone up – that is something I hear in every single forum where there are charities talking with government."

A theory of change built on evidence

The Smith Family was founded in 1922, when five Sydney businessmen told an orphanage they were “the Smiths” when asked who had anonymously donated Christmas toys to the children. For decades it operated as a broad welfare charity. But in the mid-’90s, its leaders confronted a difficult reality: team members and volunteers were meeting the children of those they had supported a generation earlier, in similar circumstances.

"I really admire the leaders at the time who reoriented the whole organisation to be about education for children and young people," Riddell says, "because that is where the strongest evidence is of breaking cycles of poverty and disadvantage."

That singular theory of change now underpins The Smith Family's flagship Learning for Life scholarship program, which supports 72,000 children and young people across Australia. The program has three components: financial assistance to cover school costs so students can participate fully; personal support from trained family partnership coordinators; and wraparound programs covering literacy, numeracy, study skills and career exposure.

Outcomes are tracked in terms of attendance, engagement and Year 12 completion, and post-school pathways such as study, vocational training and employment.

Measuring impact through ‘contribution’, not ‘attribution’

For organisations working with families over many years, isolating the cause of a good outcome is hard. A Learning for Life scholarship can follow a student from kindergarten to the end of senior secondary school. And in some families, multiple siblings are involved, for decades.

"We think about our contribution rather than attribution," Riddell said, of the tricky business of assessing whether The Smith Family’s impact is greater than other impacts. "We don't have in place an ability to say this is specifically what The Smith Family's attribution is in any of the young people who finish school and go on to have great futures. What we say is this is the contribution that we have made through supporting that young person and through their family."

The evidence base is layered: the organisation gathers quantitative data on attendance and academic achievement, supplemented by qualitative insight. The Pathways, Engagement and Transition Survey tracks 5,000 young people at a time over multiple years. Alongside the hard data, families report that the scholarship gives them a sense that someone believes in them. "I know that the Smith Family has my back" is how one family put it, Riddell said.

One of the organisation's most significant capabilities came from years of work with state education departments. Parents and carers can now consent to their child's attendance and academic data being shared directly with family partnership coordinators via a real-time dashboard. What began in South Australia has since extended to Western Australia, Queensland and Tasmania, and will soon start in Victoria.

When a student's attendance pattern changes, a coordinator can intervene before a short-term problem becomes entrenched. In one instance, a student who had stopped attending school and could not be located by the school appeared in the dashboard at a different school across the state. The family was contacted within days.

"That means we've got a really complete data set to then do our longitudinal analysis to look at how people are going over years and years," Riddell said.

Persuading different audiences about evidence

Only 15 per cent of The Smith Family's funding comes from government. The remaining 85 per cent comes from donations, often small donors, which means the organisation must make its case to multiple audiences.

"One audience who matter very, very deeply is the beneficiaries of our work," Riddell said.

"Parents care deeply about their children and they want to know, what is the benefit for my child of being part of this?"

Another is donors and funders. "What we really want to do is elevate the voice of the people who have all of the experience and showcase that in a very strengths-based way to the people who are paying for it."

Governments are giving greater credence to the evidence

Riddell points to the increasing momentum of outcomes-focused contracting, where departments articulate what “good” looks like and work with providers toward shared goals. The DSS Children and Family Support (CaFS) program, currently being designed, aims to join up client data “at the back end” so that providers can access information without all having to collect it separately, avoiding frustrating clients who in the past have been asked to undergo multiple assessment processes.

Riddell said that Treasury's "Measuring What Matters" framework – which has been publicly backed by Charities Minister and Assistant Treasurer Andrew Leigh – is another signal of the same shift towards “joined up” data.

"Theoretically, eventually, we will be able to get better insights about what works for whom and in what context without having to do all of that legwork in every single organisation and every single provider for every single contract," she said.

What other organisations can do to amplify their impact

For not-for-profit organisations without The Smith Family's scale or data infrastructure, Riddell's advice is practical.

"I don't think everybody needs to build a longitudinal data set and a full multi-year evaluation to a certain standard. Many organisations could benefit from something which is shorter term: What might we be able to observe that has changed, that allows us to create the learning loop of this is what we've observed, [so] what might we now need to do in terms of course correction based on that?"

And she suggests organisations seek out existing evaluations for similar service types. "Often there'll be someone else who has done that for the type of service that you are delivering. Being able to access somebody else's evaluation and understand what the insights were, that these were the key ingredients for success, these were some of the implementation challenges, that can tell you a lot without you having to do it yourself."

With AI, it’s still about good data

On artificial intelligence, Riddell says the quality of the underlying data will determine whether the tools will add value. "If the data is not very high quality, if it's not collected in a thoughtful and appropriate and ethical way, then it doesn't matter whether you have a human or an AI approach to trying to decipher what does this say, it won't tell us things that are useful."

What has shifted, Riddell says, is how funders now regard measurement itself. "I don't think it is any longer seen as something that's nice to have or additional. I think it's now seen as a requirement and a higher recognition that it therefore needs to be included in the budget for whatever the work is. That has shifted a lot in the last decade."

The Smith Family’s latest research

Quick ways to impove your NFP's impact

Download our FREE 42-page Measuring What Matters guide for a quick handle on evaluation principles and frameworks, outcomes and data gathering.

Then, visit the outcomes measurement section of our website for answers to '10 questions every board member needs to ask about measuring outcomes', evaluation templates, and data and privacy policies.

Find more resources in our tools and resources section.