Impact measurement: answering the five questions we come across most

_Lindsay Harrod, the FSI Consultancy team_

Why do we bother measuring impact?

We do a lot of work around ‘impact measurement’ at the FSI, and it’s certainly a buzzword for charities. But often when we talk to small charities about impact, they’re thinking of it from a pretty narrow perspective: how do we prove the value of what we do to funders?

Impact measurement should be much bigger than this. The primary motivation of measuring your impact should be strategic – to make sure your services are making the difference you hope they do, and to find ways to improve and strengthen them. If your impact plan is purely led by what funders want to know, you’ll be missing out on lots of vital information for your charity’s development.

Photo by Júnior Ferreira on Unsplash

But, we’re a tiny team: how can we create a perfect impact framework that definitively proves what we do works?

We also hear from lots of small charities that they see impact as an academic or very complicated thing, and that it’s only worth doing if it’s at an advanced or ‘best practice’ level. In fact, ‘best practice’ means sticking to one of the key Inspiring Impact principles of ‘proportionality’. That means keeping it practical and relevant to the scale of your work – in particular, proportional to the decision you’ll be making based on the findings. So, if you’re monitoring the impact of a small project in an ongoing way in order to make tweaks and improvements, then you will approach it very differently to evaluating a national scale pilot programme with many complex stakeholder groups.

“It is better to be roughly right than precisely wrong.”

John Maynard Keynes

We hope that our proportionate and collaborative approach is why the FSI’s impact training and consultancy is so popular with small charities. We keep it pragmatic and realistic, and we understand the challenges a small organisation will have in implementing an overcomplicated impact plan.

In fact, in many of our recent impact consultancy projects, we’ve ended up saving the charity time. Sometimes this is by centralising and streamlining all the many different surveys and metrics used across different projects. Sometimes it’s because we’ve held a workshop with the frontline team to understand the realities of these measures in the day to day, and they’ve suggested more practical ways to overcome any barriers. For instance, at one recent client, the leadership team was trying to get the front desk team to take every new client through a pre-service assessment form, but they were getting a lot of push back. We held a session going back to basics – what do we want to measure and why, and got feedback from the front desk team about what they thought the best way to do that would be. There was a really interesting conversation and some great ideas that are now being taken forward, and there’s now lots more buy in, even with the same capacity and time challenges.

There’s so much we could measure: how do we choose where to start?

Often doing this work saves you time because we’ve spent some time really digging into what questions you actually need to answer – what are you collecting just for the sake of it, and what do you actually need?

So, let’s start there: take a few minutes now to think about what questions you want your impact practice to answer. Think about all the different perspectives too – your board, your frontline team, your service users themselves, your supporters. In fact, we’d recommend involving these different groups in establishing your impact measurement plan – that’s another of the key principles of good impact practice!

15 key questions you might ask could include:

  1. Are most people achieving the outcomes we hope they will after 3/6/12 months?

  2. Are different groups having different experiences e.g. different needs, demographics, referral journeys?

  3. Are there any arising or unmet needs we should be aware of?

  4. Are we making assumptions about the difference we make?

  5. Can we test any of these assumptions to make sure they hold true?

  6. How confident are we that the short-term changes we see do lead to the long-term changes we are aiming for?

  7. Are there any trends or outliers that could teach us something? E.g. is the % of beneficiaries achieving an outcome increasing or decreasing, is there a big disparity between beneficiaries (some achieving a big difference and others none)

  8. Is one project having a bigger impact than another?

  9. Do beneficiaries benefit more from doing multiple projects than just one?

  10. How can we improve our services?

  11. Is there a need to expand our services or work in partnership with other organisations?

  12. What difference do I [a staff member or volunteer] make and what could I be doing even better?

  13. Are there external factors affecting the journey that beneficiaries go on?

  14. Should we be doing anything to respond to these external factors?

  15. What changes could I [a service user] expect to experience by joining this service?

From here, you can start to flesh out what information you need to collect. You might already have lots of it e.g. pre and post service assessments, feedback surveys. You might also need to undertake more research, such as focus groups or interviews.

We’ve had to completely change how we operate in COVID – how do we adapt our impact measurement?

Like most of us, you’re likely to have needed to adapt your impact measurement methods during COVID due to social distancing and also changing needs. We know it’s been really hard to do that with all the change going on, so one top tip is to take a step back and trust your gut instinct. If you as a team know you’ve been making a difference and meeting needs, from everything you’ve observed, from your conversations, and from the stories of where people are now, then that is a good place to start.

You can also listen to your gut to help you tease out the priority things to revisit. We are seeing lots of charities using the disruption of COVID-19 to refresh their Theory of Change and adapt the journey for the ‘new normal’. For instance, perhaps you suspect Project A really isn’t fit for purpose now – how can you get some information to help you identify how you can improve it? Or perhaps you’ve seen your service users’ needs change and now they also need support in another area before they can benefit from your service – how might you understand what those needs are and how you or another organisation can meet them? So, use your common sense here, and start by opening up these honest, reflective conversations with your team.

Finally, our last tip for impact practice in the current environment is to involve your service users. Open up the conversation right at the start – what are their needs, and what outcomes do they think are important? Why do these changes matter to them? What do they think of your current measurement practices? How has COVID changed things? This could be through surveys or focus groups, or it could be through anecdotal conversations. Have clear and open mechanisms for beneficiaries to give feedback and demonstrate that you value their input. For more in-depth feedback, you may want to consider paying them (as ‘experts by experience’) or establishing a more formal mechanisms such as an advisory panel. Hold them as equal partners in your work – after all, they know more than anyone what difference your service can make!

So, what support can we access?

The FSI has tonnes of support available about impact measurement, and there are lots of other sources available too: