Innovative Evaluation

Written by Cari Patterson, Director of Research and Evaluation

When we work in complexity, we are often evaluating innovation. That means we are looking at the effectiveness and impact of new ideas that are being tested. And for innovations to be tested, there has to be a space for taking risks, failing, succeeding, and most of all, learning. Now, in the COVID 19 context, in this time of rapid change and adaptation, innovation is happening so quickly that it is difficult to keep track of everything. And that makes it challenging to evaluate the impact – but it also creates an opportunity to be innovative with evaluation.

Because most people are focused on how to respond to the situation, and on how to organize supports for people who need them, they don’t have energy, time – or likely the inclination – to participate in evaluation interviews, focus groups, and surveys, some of the big ways we gather evaluation data. So we are in a situation where there is so much to capture and learn, and we have to pull some different tools out of our toolbox – so that we can capture data in creative and easy ways that don’t burden the people who are in the middle of doing the intense and important work. 

At Inspiring Communities, we are using Photovoice. We have asked all of our teams to pay attention to the helpful ways of working that are emerging at this time. We’ve asked everyone to take 1-2 photos a week that address this question. During our weekly online team gatherings, we are spending 45 minutes sharing the pictures and making sense of what they mean. Doing this activity on a weekly basis will allow us to observe changes over time. We have also invited our teams to invite their communities to participate in this activity. When we come out the other side of COVID 19, we will have lots of rich data that we can reflect on together, both in communities, and with Inspiring Communities as a whole.

And there are other ways we can be creative about collecting data during this challenging time. As evaluators, we can keep journals; we can identify a few key critical questions to keep in mind, and note our observations in journal entries. We are doing this at Inspiring Communities. Our Evaluation Team agreed on a set of questions, and we will bring our observations together in the weeks to come. You can also extend an invitation to others to journal about their observations and experiences, and let them know you will be asking them to share their insights later.

Letter-writing is another way to capture important observations that doesn’t require us to ask a lot of people who are doing the work. We can pair up with another evaluator to write letters to each other about what we are seeing, and/or we can invite people who are doing the work to exchange letters with us about their experience. 

We can use unstructured story-telling to capture information in real time. Once a week, we can ask people doing the work to tell us about the work they were involved in. We can record these conversations so we have the data in real time, and later work with the person involved in the work to pull out themes and learnings from their experience.

We can create a social media group to gather people’s responses to key questions, or their thoughts in general about what they are experiencing. Try posting a question or an observation once or twice a day , and invite people to comment.

Another idea is to use online surveys (such as Google Forms) to capture quick feedback and ideas after an online conversation – just keep it short and snappy, and make it look attractive and enticing. 

The current context offers rich potential to learn about systems change. It also offers evaluators an opportunity to be nimble and creative in the way we harvest the richness. We’d love to hear about other creative evaluation methods you are using!