Posts Tagged ‘training’

 

How to conduct a storytelling evaluation

Posted by Marc Maxson on August 2nd, 2013

Introduction

If you work at an organization and have done an evaluation before, you probably followed these steps:

  1. Define the problem
  2. Design the survey
  3. Collect data
  4. Analyze data
  5. Form a conclusion
  6. Change project direction, apply for new funding, etc.

Our process has pretty much the same steps, only in the opposite order:

  1. Analyze existing data
  2. Form a hypothesis
  3. Design a different kind of survey
  4. Collect more data
  5. Compare your new data against the existing data
  6. Form a conclusion
  7. Define the problem
  8. Change project direction, apply for funding, etc.
  9. Go to step 3; repeat.

This “different kind of survey” uses qualitative stories in a quantitative way by combining text analyiss with questions about the story in the follow-up survey. These questions help define the ambiguity hiding in our preconceptions of the problem, mapping out power relationships in communities, conflict aggressors and victims, problems, solutions, innovation, and understanding what could have happened if things had worked perfectly. We provide a distributed data collection method that ensures many voices are heard. Every response begins with a story – a brief personal narrative – and the phrases in the story are an essential part of forming a hypothesis (step 2).

  • Aggregate power: With tens of thousands of stories in the reference collection, most organizations can find hundreds of stories related to their mission among the >58,000 we already collected in Kenya and Uganda from 2010-2012.
  • Filtering: Stories “at scale” makes it possible to validate the numbers and filter out misleading information.
  • Exploration: Broadness of the narrative prompting question and the scope of indivual perspectives we’ve already heard from allows the analysis tools to reveal unexpected connections between issues, peoples, locations, and organizations.
  • Benchmarking: Another major advantage is that our survey data can be combined with results from surveys conducted by others, as well as with your previous survey data. The analysis tools are built for pooling data and comparing subsets to each other, so that data trends become more reliable over time.
  • Durable, perenial data: Traditional evaluations usually create a new survey each time or lack the technology to merge new results with old ones done by others. Our story form builder is a compromise between design flexibility and instant benchmaring. And because our design makes it easy for you to change your questionnaire while adding to your growing data collection, you can form conclusions based on all of the data (past and present!).

Analyzing existing data

Note: there is a webinar recording for this tutorial. You can access it via AnyMeeting.com:

storytelling tools webinarWebinar: Storytelling tools explained (with Britt Lake and Marc Maxson) July 2013

Our process as a series of 9 steps. This tutorial covers the first three steps:
(1)    Analyze existing data — use the story search and story phrase bubbles tools
(2)    Form a hypothesis — based on what you see
(3)    Design a different kind of survey — story form builder
(4)    Collect more data — attract a group of young people, train them, given to papers, and send them out
(5)    Compare your new data against the existing data — in the same analysis tools as step (1)
(6)    Form a conclusion
(7)    Define the problem — informed by a new perspective
(8)    Change project direction, apply for funding, etc.
(9)    Go to step 3; repeat.

Explore the story search tool at djotjog.com/search/

wpid4265-media_1374868383219.png

Go to djotjog.com/search/ and either type in a word or phrase (in quotes, like “school fee”) to see stories that contain those words. Or you can click on one of the suggested search links below.

Build a good topical story collection with at least 200 stories.

wpid4266-media_1374869022412.png

I entered some words and phrases that relate to women entrepreneurs. But note that the word “entrepreneurs” was only found in ~50 stories (out of over 58,000). Instead, words like “micro loan” and “start business” are better choices. You can add several phrases and it will combine results for all stories that contain any one of those phrases.

Example: “micro loan” “start business” “started business” entrepreneur= 159 stories with any one of these phrases
Example: ugali bread “grow food” maize rice = 1485 stories with any one of these grain crop food words
Example: women and group and business = 290 stories with all of these words

Note that there is no way to combine words with AND and OR at this time, but you can usually find hundreds of stories with the existing search.

Understanding the visual summary icons

wpid4267-media_1374869238166.png

When hundreds of stories match, the search results include a visual summary of who told these stories and what they talked about. The icons of girls, boys, women and men will appear different sizes and colors depending on how prevalent that demographic group is in the group of matching stories, compared to the number of people of the same age/sex in all 58,000 stories.

A larger icon means that the demographic group is overrepresented.
A green icon means stories from this group are more positive in tone than the overall sample.
Red means more negative.
Yellow means about the same as the whole set of stories.

Note that you can mouse over these icons and see absolute numbers for the percent of stories that came from each demographic group, or relate to one of the ten story themes.

When is a trend in the icons important?

wpid4274-media_1375125428905.png

That is a difficult question to answer because it depends on the context. What are YOU trying to learn? If you have no questions about what your community thinks and feels about a subject, and no doubts about the work you are doing, this tool is not for you. But for me, speaking as a scientist, I am constantly asking myself if I am doing the right thing for the community. This reality check helps.

In this example, the overall sample of stories that mentioned “election” (n=724) was negative. But those about recent events in the last six months (from when the story was collected) from people who felt they played a role in the story were a mix of positive and negative. By selecting a more recent timeframe I was able to see that the trend is towards more examples of respect growing (green, positive icon), yet actual freedom remains negative.

If you were wondering how I was able to reduce the 724 stories to just 24 interesting ones, I was using power search, explained next.

Use Power Search to filter stories

wpid4268-media_1374871465342.png

Check boxes next to the things you want to see in your stories

wpid4269-media_1374871612372.png

By default, the checkbox for the words you searched for will be selected. Uncheck it if you want to lookat all the stories.
Check other boxes to narrow stories by outcome, demographic group, point of view, or related topics, and git [Generate a custom report now]

Note that within a set of answers to the same question, such as type of story, you can check more than one box and it will return MORE stories than checking just one box, because it will match ANY of the answers you select.

Power search lets you filter by any criteria in the questionnaire and export as CSV.

wpid4270-media_1374872003791.png

In the example shown, 1485 stories related to types of crops people grow in Kenya have been reduced to 55 stories using filters. The POV (point of view) filter excludes stories that sound like they are from the organization’s perspective (and not a personal one). The story needs to be focused on a problem (not a solution) and about economic opportunity (not social relations or physical well being).

A quick scan of the visual summary reveals that older men tell more possible “success” stories and women share more negative stories. The themes appear to focus on food and shelter, and to a lesser extent, on self esteem. Oddly, several categories are far less represented or absent: freedom, creativity, knowledge, respect, and family.

Export as CSV

wpid4271-media_1374874022827.png

If you want to analyze these stories in any other way, feel free to grab them.

What the export contains: Everything!

wpid4272-media_1374877136551.png

age_of_story_when_told, city, country, date_transcribed, felt, group_in_story, id, latitude, longitude, need_prob_soln, net_heirarchy_topic_score, num_stories_told, org, org_type, original_organization_name, other_information, outcome, pov, pov_avg, project_focused, quality_score, revised_organization_name, sex, soc_phys_econ, story, story_char_length, story_connection, storyteller_age, storyteller_contact_sms, title, topic_creativity, topic_family, topic_food, topic_freedom, topic_fun, topic_knowledge, topic_physical_needs, topic_respect, topic_security, topic_self_esteem, translated, url, who_benefited,

You can compare a story to other stories told by the same person

wpid4273-media_1374877402221.png

This is a “within-subjects” comparison – a quick way to find out if you are deceiving yourself about how much diversity is in the viewpoints you are listening to. If a person only talks about one issue, and nothing else, they are probably not giving the full story, and possibly not the most honest one.

Summary of hypothesis generating tools

wpid4275-media_1375127124517.png

You explore a large collection of anecdotes in order to get a feel for many perspectives on around an issue. Here are some strategies, with case studies to follow:
(1) Pick a general search phrase (e.g. “street child”) and then change the filter options around demographics and point of view. Save screen shots for each perspective and look at them side by side.

(2) Build a set of stories exactly related to your work, then broaden your search and look at how the trends change as you adjust your search words.

wpid4276-media_1375127625824.png

As you can see, choosing words that are more likely to be used by regular people will lead to stories that have less positive bias. Stories from farmers and that use the NGO phrase “food security” are more positive than those about “grow food” and bread, ugali, etc. — yet stories about these foods are true “food security” stories.

(3) Compare stories related to your work with any other topics that matter to these same people

wpid4277-media_1375128434560.png

In this example – over 5000 stories related to farming and agriculture and “food security” there is a reference group of 1106 stories NOT about these subjects but shared by people who gave these stories. The ability to do a “within-subjects” comparison on the fly is unique to this approach. We will be building more tools that make this easier to visualize. For now, you can begin by browsing these stories.

Bubbles: A more abstract comparison tool

wpid4278-media_1375128634704.png

The “bubbles” tool attempts to merge and parse all of the narratives in a single view. Words that are common and interesting appear in bubbles. The location of the bubble above or below a line is determined by the meta-data for each story. So far, the types of comparisons this tool allows are:
(1) is the story related to a known NGO or is it some unmet need, (or work by people not NGOs?)
(2) comparing success vs failure stories
(3) how does the number of stories a person shares relate to what is said?

Our process as a series of 9 steps. This tutorial covers the third step:
(1)    Analyze existing data — use the story search and story phrase bubbles tools
(2)    Form a hypothesis — based on what you see
(3)    Design a different kind of survey — story form builder
(4)    Collect more data — attract a group of young people, train them, given to papers, and send them out
(5)    Compare your new data against the existing data — in the same analysis tools as step (1)
(6)    Form a conclusion
(7)    Define the problem — informed by a new perspective
(8)    Change project direction, apply for funding, etc.
(9)    Go to step 3; repeat.

Go to GlobalGiving.org, log into your project leader account, then click on the ‘stories’ button on your dashboard

media_1375302753936.png

If you are not currently a project leader you can use this secret link: http://www.globalgiving.org/dy/v2/storyInput/info to collect stories, but you should join officially. The GlobalGiving Organization application page is at http://www.globalgiving.org/non-profits/join-globalgiving/

Read the stories page overview

media_1375303268841.png

Understand the “scribes” model for collecting stories before you build your form

media_1375373831150.png

(1) Engage a group of local people. We find that girls who have just graduated from high school do the best work, and provide you with a broad sample of both men and women, both young and old.
(2) Invite them to a 1 hour training. You want to end up with about a dozen, but you may have to invite 20 or 30 as some will not be interested.
(3) Have them fill out the one-page story form you are building during the training.
(4) Give them blank printed story forms and a goal: interview 10 people in the next 2 weeks. Ask each one for 2 stories. And return all the papers.
(5) Provide them with an incentive to participate – this can be a small payment (in East Africa we found that 10 cents per story was sufficient) or a privilege, or additional skills training.
(6) Collect the stories and scan them using an Android camera phone or iPhone.
(7) Analyze, and share your findings with the scribes. They will want to know what the community is talking about. Then repeat this every 6-12 months.

Customize your own storytelling form. The first questions are selected for you. They are required.

media_1375389676208.png

You will see that two of the required questions are scribe’s phone number / email address and the storyteller’s phone number / email address. These will be kept confidential and are used for tracking how many people are actually filling out the forms, and which stories were told which whom. We prefer that you tell your scribes to choose one kind of data (phone OP email) and ask for that one kind of information for everybody.
If somebody (i.e. a scribe in training) tells his or her own story and there is no scribe involved to write it or collect it, then put the same phone number in both boxes.

The next questions, in yellow, are optional. You may add them to your form by clicking on the checkboxes to the right.

media_1375389791695.png

Which questions you choose is up to you. It will NOT allow you to add more than front and back of one page worth of questions.

Each of the optional questions are best used with one or more other questions in the set, depending on the kind of stories that interest your organization.

media_1375452473656.png

The choices may seem overwhelming at first, so below are a few examples of an organization with a specific goal and which questions they chose.

Example: Story-based program monitoring for a girls’ after school program

media_1375453302330.png

Vijana Amani Pamoja runs the mrembo project in Kamukunji, a Nairobi slum. Mrembo (“beautiful girl” in swahili) aims to give adolescent girls ages 8-15 life skills for deal with boys and other threats in their daily lives. This organization collects stories from girls before and after each 10-week session to keep track of what issues are coming up in their girls’ lives that need to be addressed in future sessions.

I’ve highlighted the optional questions in red. This organization chose to focus on the who, what, and why of the story, along with two questions about power and hierachy.

Example: VSO’s Valuing Volunteering community research project questionnaire

media_1375453875639.png

Two VSO volunteers (Jody in Philipinnes and Simon in Kenya) are gathering community feedback about what would make volunteerism more effective in developing countries. They’ve included a lot more of the conflict and power mapping questions, and some less structured open-text questions, like “what else would have made a difference in this story?”

Even though few of these questions overlap with VAP, both will be able to benchmark their answers against the other on these questions and story elements:
1. All words in the actual story — you build a set of relevant stories by searching the text first.
2. topics
3. age
4. sex
5. role
6. authority figures
7. why it happened
8. who benefitted (outcome)
9. types of solutions

That’s nine different criteria even though these two groups have never coordinated on which questions they’ll ask. And there are many more people collecting, so that everyone will eventually have comparable data for every question they could ask.

Choose your questions and submit the form.

media_1375454317887.png

You can mouse over the tooltip (?) symbol for more information about the context for each question.

Your form is ready – check your email for instructions

media_1375454484521.png

The email will contain unique URLs to your specific story form. There are two links. One is a web form where you can enter data. Our free transcription service will use this when you scan your stories with a camera phone and email them to us (marcmaxson@globalgiving.org or projecthelp@globalgiving.org). The other link is a printable PDF that you should print and photocopy many times so that scribes can collect stories anywhere without needing any technology.

Scanning stories with a smart phone

media_1375455627042.png

We have an instructional video to demonstrate this. Basically, there are many android and iPhone apps that do repetitive document scanning faster than a conventional scanner. We recommend you download one of these:

Scan to PDF (free on android)
Genius Scan (free on iPhone, $2 on android)

These apps will create a PDF and attach it to a gmail message for you. If you address it to us at GlobalGiving, we will do the rest and email you back when your group of stories is available online for analysis (at djotjog.com/search/ and djotjog.com/bubbles/)

Ready to begin?

— marc maxson (mmaxson at globalgiving dot org)

Tips & Tools for NGO Financial Management

Posted by Alexis Nadin on October 20th, 2010

It’s no secret: managing a non-profit’s finances is tough. Non-profit organizations are pushed to think creatively about how to do more with less.  Sources of income are inconsistence and donations come with strings attached.

On October 20, 2010, GlobalGiving hosted two webinars on Financial Management presented by Denise Phelps, the Treasurer of Maryland School for the Deaf and Civitan Club. During this one-hour training, Denise shared tips and tools for managing an organization’s finances including: (more…)

Zero to Social: GlobalGiving’s Social Media Trainings

Posted by Alexis Nadin on October 5th, 2010

Social isn’t just a tactic for marketing; social is how we live. And as a nonprofit, it should be how your supporters give. Social giving only helps your organization by letting your biggest fans be your advocates. Social networks are tools you can use with your overall marketing strategy to make sure you stay relevant to your supporters.

On September 29, 2010, Alison McQuade, GlobalGiving’s Online Marketing Manager and social media guru, hosted beginner and intermediate trainings on social media. (more…)