This 3-part series reflects on GlobalGiving’s experiments testing the findings of The Narrative Project. The Narrative Project was a wide-scale research effort to improve the public perceptions of global development in donor countries. Part one of our series explains GlobalGiving’s research methods and findings. Image from NarrativeProject.org.
In the summer of 2015, I was one of ten nonprofit communicators who received a grant to test the findings of The Narrative Project. The Narrative Project was a wide-scale research project driven by the Bill & Melinda Gates Foundation, InterAction, and other major NGOs, that aimed to increase public perceptions of global development in the US, UK, France, and Germany. The Narrative Project researchers found that messages and stories carrying certain narrative themes—independence, shared values, partnership, and progress—motivated certain segments of the population to change their perceptions of global development. It also found an increase in the target group’s self-reported likelihood to take action to support global development causes. I was interested in whether or not Narrative Project themes would motivate people not only to change perceptions but also to act: to donate to global development projects.
In my work at GlobalGiving, I facilitate storytelling and fundraising training for thousands of nonprofits of all sizes, all around the world. So I wanted to find out how easy it was for nonprofits to adopt the Narrative Project recommendations, and, more importantly, to find out if nonprofits who used it would raise more money in their online fundraising appeals.
Our summer of experimentation looked like this:
1. Diving into Data: First, my GlobalGiving colleagues and I used the guides provided by the Narrative Project to create a natural language processing tool that would score a piece of text based on how well it aligned to The Narrative Project Themes. We then ran more than 50,000 ‘project reports’ (or stories) written by our nonprofit partners in 160+ countries over the past eight years through the tool to find highly-aligned (and non-aligned) reports.What we found was that the Narrative Project-aligned reports triggered a statistically-significant lower number of average donations than the other reports, but the effect size was small. So, according to our study of eight years of Narrative Project-aligned reports, stories told with the Narrative Project themes always performed slightly worse in terms of fundraising.
2. Practicing Before We Preached: Prior to spreading the gospel of the Narrative Project with our nonprofit partners, we wanted to find out if using it in our own fundraising appeals would raise more money. Over a period of three months, we sent out more than half-a-million emails that were part of six A/B tests. We’d pick a topic, write two versions of the appeal (we’d first write the “A” text and then modify it to include Narrative Project themes in the “B” version), and then we’d split our recipient list, and send each version to one half of the list. Here are examples of an A (control) and B (test) version of an appeal we sent. After the six A/B tests, we found that the control text (the non-Narrative Project wording) performed, on average, 117% better. In other words, The Narrative Project language didn’t help raise more funding support for nonprofits, in fact, it raised less than the control.
3. Training and Testing: We lead a one-hour session during our Social Impact Academy to talk about social impact communications with 50 locally-driven nonprofit organizations from 16 different countries. We asked each of the participants to submit a written project report containing a story prior to the training. After the training, we asked them to revise the story based on what they learned. We scored the before and after texts, and asked participants for feedback about the training. What we learned was this: first, it was difficult to convey the principles of the Narrative Project with a single training in a way that significantly changed our partners’ writing. On average, our partners’ report quality (and Narrative Project alignment) only increased by 5% as graded with a scoring tool developed from the User Guide. (This means they were only marginally able to adopt the Narrative Project recommendations after a brief training and introduction to the User Guide.) However, this session was the highest-rated of our nine Social Impact Academy trainings, demonstrating that there is an appetite among nonprofits to learn how to drive more social impact (and funding) with different methods of storytelling.
We weren’t able to find any circumstances, either in our historical analysis of reports written by others, or in our own A/B testing of Narrative Project text, where the Narrative Project themes actually moved potential donors to take immediate action to fund global development projects. In fact, we found the opposite to be true in most cases.
I suspect there were few people out there who wanted our tests of The Narrative Project to ‘work’ more than me. But I realize now the irony in my optimism; I don’t believe that there’s one ‘silver bullet’ for solving the complex problem of poverty, why would I believe there is one easy solution for all communicators, whose work is nuanced, creative, and complex?
Instead, we should see The Narrative Project research as fuel for an ongoing conversation about better storytelling in global development. In my next two articles I’ll share more about how the Narrative Project re-framed the way I think about “what works” for communicators in Global Development, some hypotheses about why the Narrative Project didn’t work at least in our experiments, and, finally, what our research demonstrates might work better instead.
Alison Carlman is the Director of Marketing and Communications at GlobalGiving. This is the first part of this three part series. Please email us at email@example.com with any questions!