This week of the Social Impact Academy, we were joined by three panelists that shared insights into their organizations’ impact assessment methods. These organizations have built distinct impact measurement approaches into their programs in ways that meet the specific needs of the communities they serve.
Session Recording: http://www.meetingburner.com/b/globalgiving/view_recording?c=9EJUJV&h=f
Joining us on the panel were:
Sandeep Ahuja, Founder and CEO, Operation ASHA
Operation ASHA provides last-mile medical services in India and Cambodia by employing local community members to improve the health of disadvantaged patients. One of its strengths is using technology to reduce costs.
Emma Pfister, Global Communications and Marketing Manager, Water for People
Water for People works in Central America, South America, Africa, and India to provide safe and permanent water sources for communities. Their impact is measured over time by assessing the increase in access to water and the likelihood that access is sustainable.
Elizabeth Ross, Director, Kasiisi Project
Kasiisi Project operates in Kibale National Park, Uganda. It is a conservation and education program with 14 rural primary schools. Its fundraising partners are in the UK and US and it functions on a small budget.
What methods do you use to assess your impact? What systems do you use to streamline the measurement process?
Use technology to streamline the process.
- Akvo FLOW: We want measurement and analysis to be quick! Akvo FLOW is an app for handheld devices that are used in the field and collects information from surveys and automatically computes analytics.
- Everyone Forever Tracker: We want a way to share it! The EF Tracker transforms our analyzed data and shows it in a consumer friendly way. The analytics become visually appealing and easy for any type of donor to understand.
- Reflection Sessions: All of our offices do annual in-country conferences to reflect on the past year. We use this as a building platform to learn and move forward.
- What gets measured gets done.Measurement of outcomes is an integral part of our methodology. Our management information systems include continuous monitoring and rigorous reporting.
- Technology: Operation ASHA uses technology to improve performance, increase productivity, and reduce costs. Use of technology reduces the cost of paper, labor, and statistical analysis.
- Use available data. Our major goal is to improve academic performance. We measure academic performance using government administered primary leaving exams. We collect this data every year for our schools and evaluate it through comparison to other schools.
- Use pre and post questionnaires. For our outside-of-school programs we use questionnaires with pre and post intervention. Changes in knowledge and attitude are noted through simple methods such as counting a show of hands after a question.
- Use one evaluation for multiple grants to avoid survey fatigue because every grant we receive requires evaluations. We also keep the same questions for multiple years so that we can keep records for long-term comparison results.
What metrics and indicators do you measure? How did you decide to use these indicators?
- Incorporate measurement from the start. Our program and measurement have never been considered two separate activities. We go to the beneficiaries, ask what they want, and develop a set of measureable indicators based on their answers.
- Data analysis. Our medical electronic recording system allows us to see analytics from year to year to measure if we are making a difference. We track number of TB symptomatics found, the number of people served, and treatment success.
- Match your indicators to your organization’s goals. Our indicators address issues concerning clean water for everyone forever within a community. Example metrics include water service level, frequency of pipe damages, and ability of community committees to pay for repairs on the water systems.
- Match your evaluations to the community you serve. For example, we use the children’s drawings as evaluations. We want to make activities fun so children stay engaged in the evaluation process, especially since they are run outside of school.
- Note what kind of analysis you want to perform in the future. We tend to use true/false questions with various programs because we want numbers for statistical assessment.
What staff and financial resources are necessary to measure impact?
- Improving our measurement is seen as investment. Measurement can’t be separated from performance, it is part of our process. We do not spend more than 10% of our operating budget on measurement.
- Technology. We realized that we could not run our program efficiently and at a low cost unless we used technology. We put together a software requirement specification proposal and sent it to Microsoft Research. Microsoft provided us with a free medical electronic recording system.
- Get monitoring embedded in local institutions. Your organization, the community, and the local government should all be part of the reporting process. This allows you to see where gaps occur in your effectiveness.
- Track how much you and your partners spend on monitoring so that you can accurately forecast your costs.
- Use employees. We only use our permanent staff for measuring impact. We design our evaluations with staff. This ensures that teachers can appropriately explain the evaluation so that the children understand them. You must have experienced people to understand children’s answers as well. We have low costs by way of materials, but about 10-15% of our budget is spent on monitoring and evaluation.
How do you involve the community you serve in the process of measuring your impact?
- Involving the community ensures quality results. We measure parameters that are the most important to our patients and communities and those who deliver the outreach program are from within the community. Our program is flexible to move according to the needs of patients, technology, and realities on the ground.
External monitoring. Auditors visit health centers at random and ask patients of those centers questions about the quality of their care. This ensures that the community is a part of our measurement process.
- Reflection sessions. After data is analyzed from the annual Akvo FLOW surveys, partners, governments, NGOs, and the community come together to reflect on the data. We discuss the problems and plan for the future together.
- We involve the community from the start. We ask what parents want their children to know and what area authorities think kids should learn about. The community helps to write evaluation sheets.
- Share the results. We present our findings to teachers, with parents, at churches, etc.