This week of the Social Impact Academy, we were joined by three panelists that shared insights into their organizations’ impact assessment methods. These organizations have built distinct impact measurement approaches into their programs in ways that meet the specific needs of the communities they serve.
Session Recording: http://www.meetingburner.com/b/globalgiving/view_recording?c=9EJUJV&h=f
Joining us on the panel were:
Sandeep Ahuja, Founder and CEO, Operation ASHA
Operation ASHA provides last-mile medical services in India and Cambodia by employing local community members to improve the health of disadvantaged patients. One of its strengths is using technology to reduce costs.
Emma Pfister, Global Communications and Marketing Manager, Water for People
Water for People works in Central America, South America, Africa, and India to provide safe and permanent water sources for communities. Their impact is measured over time by assessing the increase in access to water and the likelihood that access is sustainable.
Elizabeth Ross, Director, Kasiisi Project
Kasiisi Project operates in Kibale National Park, Uganda. It is a conservation and education program with 14 rural primary schools. Its fundraising partners are in the UK and US and it functions on a small budget.
What methods do you use to assess your impact? What systems do you use to streamline the measurement process?
Use technology to streamline the process.
- Akvo FLOW: We want measurement and analysis to be quick! Akvo FLOW is an app for handheld devices that are used in the field and collects information from surveys and automatically computes analytics.
- Everyone Forever Tracker: We want a way to share it! The EF Tracker transforms our analyzed data and shows it in a consumer friendly way. The analytics become visually appealing and easy for any type of donor to understand.
- Reflection Sessions: All of our offices do annual in-country conferences to reflect on the past year. We use this as a building platform to learn and move forward.
- What gets measured gets done.Measurement of outcomes is an integral part of our methodology. Our management information systems include continuous monitoring and rigorous reporting.
- Technology: Operation ASHA uses technology to improve performance, increase productivity, and reduce costs. Use of technology reduces the cost of paper, labor, and statistical analysis.
- Use available data. Our major goal is to improve academic performance. We measure academic performance using government administered primary leaving exams. We collect this data every year for our schools and evaluate it through comparison to other schools.
- Use pre and post questionnaires. For our outside-of-school programs we use questionnaires with pre and post intervention. Changes in knowledge and attitude are noted through simple methods such as counting a show of hands after a question.
- Use one evaluation for multiple grants to avoid survey fatigue because every grant we receive requires evaluations. We also keep the same questions for multiple years so that we can keep records for long-term comparison results.
What metrics and indicators do you measure? How did you decide to use these indicators?
- Incorporate measurement from the start. Our program and measurement have never been considered two separate activities. We go to the beneficiaries, ask what they want, and develop a set of measureable indicators based on their answers.
- Data analysis. Our medical electronic recording system allows us to see analytics from year to year to measure if we are making a difference. We track number of TB symptomatics found, the number of people served, and treatment success.
- Match your indicators to your organization’s goals. Our indicators address issues concerning clean water for everyone forever within a community. Example metrics include water service level, frequency of pipe damages, and ability of community committees to pay for repairs on the water systems.
- Match your evaluations to the community you serve. For example, we use the children’s drawings as evaluations. We want to make activities fun so children stay engaged in the evaluation process, especially since they are run outside of school.
- Note what kind of analysis you want to perform in the future. We tend to use true/false questions with various programs because we want numbers for statistical assessment.
What staff and financial resources are necessary to measure impact?
- Improving our measurement is seen as investment. Measurement can’t be separated from performance, it is part of our process. We do not spend more than 10% of our operating budget on measurement.
- Technology. We realized that we could not run our program efficiently and at a low cost unless we used technology. We put together a software requirement specification proposal and sent it to Microsoft Research. Microsoft provided us with a free medical electronic recording system.
- Get monitoring embedded in local institutions. Your organization, the community, and the local government should all be part of the reporting process. This allows you to see where gaps occur in your effectiveness.
- Track how much you and your partners spend on monitoring so that you can accurately forecast your costs.
- Use employees. We only use our permanent staff for measuring impact. We design our evaluations with staff. This ensures that teachers can appropriately explain the evaluation so that the children understand them. You must have experienced people to understand children’s answers as well. We have low costs by way of materials, but about 10-15% of our budget is spent on monitoring and evaluation.
How do you involve the community you serve in the process of measuring your impact?
- Involving the community ensures quality results. We measure parameters that are the most important to our patients and communities and those who deliver the outreach program are from within the community. Our program is flexible to move according to the needs of patients, technology, and realities on the ground.
External monitoring. Auditors visit health centers at random and ask patients of those centers questions about the quality of their care. This ensures that the community is a part of our measurement process.
- Reflection sessions. After data is analyzed from the annual Akvo FLOW surveys, partners, governments, NGOs, and the community come together to reflect on the data. We discuss the problems and plan for the future together.
- We involve the community from the start. We ask what parents want their children to know and what area authorities think kids should learn about. The community helps to write evaluation sheets.
- Share the results. We present our findings to teachers, with parents, at churches, etc.
Q: Elizabeth, how do you use drawings for depth (complex answers)?
Elizabeth: Here is an example. Our water project takes kids into the field to measure water quality and learn about environmental degradation. We have kids draw a picture of a pond before and then after the field trip. We know the program is affecting them because we can compare their understanding of agricultural impact on water quality based on the pictures’ content.
Q: Elizabeth, what do you mean when you use one single evaluation for multiple projects?
Elizabeth: We both create one evaluation and use it multiple times, and we use one evaluation for multiple projects. For example, we do conservation education through wildlife clubs which are funded by Disney. Then we have more specific projects like building empathy with chimpanzees which are funded by others. What we want to know from both programs is whether there is a change in the children’s attitudes. We measure this with an attitude test; the same evaluation is used for both programs.
Q: Could you provide an example of a project design that included impact from the start?
Elizabeth: I would just say that you have to. Our granting bodies require us to do that. And you must know what you are going to measure ahead of time for statistical analysis.
Emma: Define what impact means for your organization and anything you do should reflect that. Look at your impact results and learn along the way. Find the changes you need to make to improve effectiveness.
This post was written by Kennan Howlett, Program Team Intern.