Social Impact Academy Posts

Social Impact Academy: Building Impact Measurement into Programs

Social Impact Academy: Building Impact Measurement into Programs

This week of the Social Impact Academy, we were joined by three panelists that shared insights into their organizations’ impact assessment methods. These organizations have built distinct impact measurement approaches into their programs in ways that meet the specific needs of the communities they serve.

Session Recording: http://www.meetingburner.com/b/globalgiving/view_recording?c=9EJUJV&h=f

Session Summary:

Joining us on the panel were:

Sandeep Ahuja, Founder and CEO, Operation ASHA

Operation ASHA provides last-mile medical services in India and Cambodia by employing local community members to improve the health of disadvantaged patients. One of its strengths is using technology to reduce costs.

Emma Pfister, Global Communications and Marketing Manager, Water for People

Water for People works in Central America, South America, Africa, and India to provide safe and permanent water sources for communities. Their impact is measured over time by assessing the increase in access to water and the likelihood that access is sustainable.

Elizabeth Ross, Director, Kasiisi Project

Kasiisi Project operates in Kibale National Park, Uganda. It is a conservation and education program with 14 rural primary schools. Its fundraising partners are in the UK and US and it functions on a small budget.

What methods do you use to assess your impact? What systems do you use to streamline the measurement process?

Emma:

  • Use technology to streamline the process.

    • Akvo FLOW: We want measurement and analysis to be quick! Akvo FLOW is an app for handheld devices that are used in the field and collects information from surveys and automatically computes analytics.
    • Everyone Forever Tracker: We want a way to share it! The EF Tracker transforms our analyzed data and shows it in a consumer friendly way. The analytics become visually appealing and easy for any type of donor to understand.
    • Reflection Sessions: All of our offices do annual in-country conferences to reflect on the past year. We use this as a building platform to learn and move forward.

Sandeep:

  • What gets measured gets done.Measurement of outcomes is an integral part of our methodology. Our management information systems include continuous monitoring and rigorous reporting.
  • Technology: Operation ASHA uses technology to improve performance, increase productivity, and reduce costs. Use of technology reduces the cost of paper, labor, and statistical analysis.

Elizabeth:

  • Use available data. Our major goal is to improve academic performance. We measure academic performance using government administered primary leaving exams. We collect this data every year for our schools and evaluate it through comparison to other schools.
  • Use pre and post questionnaires. For our outside-of-school programs we use questionnaires with pre and post intervention. Changes in knowledge and attitude are noted through simple methods such as counting a show of hands after a question.
  • Use one evaluation for multiple grants to avoid survey fatigue because every grant we receive requires evaluations. We also keep the same questions for multiple years so that we can keep records for long-term comparison results.

What metrics and indicators do you measure? How did you decide to use these indicators?

Sandeep:

  • Incorporate measurement from the start. Our program and measurement have never been considered two separate activities. We go to the beneficiaries, ask what they want, and develop a set of measureable indicators based on their answers.
  • Data analysis. Our medical electronic recording system allows us to see analytics from year to year to measure if we are making a difference. We track number of TB symptomatics found, the number of people served, and treatment success.

Emma:

  • Match your indicators to your organization’s goals. Our indicators address issues concerning clean water for everyone forever within a community. Example metrics include water service level, frequency of pipe damages, and ability of community committees to pay for repairs on the water systems.

Elizabeth:

  • Match your evaluations to the community you serve. For example, we use the children’s drawings as evaluations. We want to make activities fun so children stay engaged in the evaluation process, especially since they are run outside of school.
  • Note what kind of analysis you want to perform in the future. We tend to use true/false questions with various programs because we want numbers for statistical assessment.

What staff and financial resources are necessary to measure impact?

Sandeep:

  • Improving our measurement is seen as investment. Measurement can’t be separated from performance, it is part of our process. We do not spend more than 10% of our operating budget on measurement.
  • Technology. We realized that we could not run our program efficiently and at a low cost unless we used technology. We put together a software requirement specification proposal and sent it to Microsoft Research. Microsoft provided us with a free medical electronic recording system.

Emma:

  • Get monitoring embedded in local institutions. Your organization, the community, and the local government should all be part of the reporting process. This allows you to see where gaps occur in your effectiveness.
  • Track how much you and your partners spend on monitoring so that you can accurately forecast your costs.

Elizabeth:

  • Use employees. We only use our permanent staff for measuring impact. We design our evaluations with staff. This ensures that teachers can appropriately explain the evaluation so that the children understand them. You must have experienced people to understand children’s answers as well. We have low costs by way of materials, but about 10-15% of our budget is spent on monitoring and evaluation.

How do you involve the community you serve in the process of measuring your impact?

Sandeep:

  • Involving the community ensures quality results. We measure parameters that are the most important to our patients and communities and those who deliver the outreach program are from within the community. Our program is flexible to move according to the needs of patients, technology, and realities on the ground.
  • External monitoring. Auditors visit health centers at random and ask patients of those centers questions about the quality of their care. This ensures that the community is a part of our measurement process.

Emma:

  • Reflection sessions. After data is analyzed from the annual Akvo FLOW surveys, partners, governments, NGOs, and the community come together to reflect on the data. We discuss the problems and plan for the future together.

Elizabeth:

  • We involve the community from the start. We ask what parents want their children to know and what area authorities think kids should learn about. The community helps to write evaluation sheets.
  • Share the results. We present our findings to teachers, with parents, at churches, etc.

Social Impact Academy: Impact Assessment Methods and Tools

Social Impact Academy: Impact Assessment Methods and Tools

Jennifer Lentfer, Director of Communications at IDEX, joined us for an overview of impact measurement for the fifth session of the Social Impact Academy. We discussed several assessment tools and methods that can make your organization more effective.

Session Recording: http://www.meetingburner.com/b/globalgiving/view_recording?c=HP7UJJ&h=f

Session Summary:

Six ways to improve your organization’s credibility

  1. Know your audience
  2. Scale back the flowery language
  3. Back up your claims
  4. Cite your sources
  5. Show, don’t tell
  6. Be honest about what you know and what you don’t know

What makes evidence credible? Use the three classical modes of persuasion.

  1. Logos: Facts and figures, proof of a causal relationship
  2. Pathos: Emotional appeal, stories of impact have deep implications
  3. Ethos: Legitimacy and credibility, convince your audience to trust you and take action

Legitimize your stories with data and humanize your data through emotional response.

What can we expect along the way to social impact? Breakdown your roadmap.

  • Problem analysis: What are we trying to change?
    • Baseline: A clear understanding of the situation before the project began
  • Goals, objectives: Where do we want to go?
  • Strategy, activities: How are we going to get there? Ex. School feeding
  • Results
    • Output: Have the activities taken place? Ex. Children received more nutritious food
    • Outcome: Did the change happen? Ex. Children’s weight and height increased
    • Impact: Was there change at community/societal level? Ex. Healthier children
    • Indicators and targets: How do we know we are on the right road?

Tools to Organize Impact Assessments

Road-mapping tools can help to define how you expect to reach your social impact goal. Below are two examples of road-mapping tools. Both provide visual approaches to help you organize the planning, implementation, monitoring and evaluation of projects.

  1. Logical Frameworks, or Logframes, break down the pathway to social impact in a chart. It defines your goals, activities, and subsequent actions.
  2. Theories of change are another way to visually track your pathway to social change. It focuses on defining long-term goals and mapping backward to meet preconditions. This method is useful because it shows that plan-making is not a linear process as the logframe might suggest.

Choosing measurement methods

Keep cost and complexity in mind as you develop your plan; the more legitimate your methods are, the more resources it will require.  Below are a few examples of measurements which span from the lowest cost and complexity (observations), to the highest (randomized control trials).

  • Observation and routine recordkeeping: This usually involves things you are already doing! It is useful for documentation of specific activities. Be sure to make your observations systematic.
  • Key informant interviews: This research technique is more than a normal conversation. It involves in-depth interviews used to collect information from a range of people.
  • Focus group discussions: These explore the motivations and perceptions of a homogeneous group. It is useful because there is a larger number of people offering insights and you can quickly collect complex qualitative information.
  • Surveys: Be careful when administering surveys, many groups are over-surveyed. Surveys are useful to present statistically accurate data to make correct generalizations.
  • Randomized control trials: This involves having a control group and an experimental group. It is especially valued because it eliminates bias. It is the most legitimate and credible measurement method but certainly not feasible for everyone.

Data collection tips

  • Use indicators to mark the progress you are making. Example indicators include a percentage of the target population, an average, a score in an index, a ratio, and the presence/absence of a condition.
  • Do not collect more data than you can analyze, plan how the data will be used so you don’t waste resources.
  • Record keeping matters! Always date documents and files.
  • Integrate your measurements in day-to-day work.
  • Be a learning organization. Take data, internalize it and shift your organizational structure to learn from the past and plan for the future.
  • Indicators should be comparable. Use baselines for this, i.e. “Before our project started, this was the situation. Now this is the situation.”

Social Impact Academy: Listening and Responding to Community Feedback

Social Impact Academy: Listening and Responding to Community Feedback

We were joined on Wednesday by David Bonbright from Keystone Accountability, Sophie Sahaf from LIFT, and Sarah Hennessy from Feedback Labs for a discussion on community feedback in the fourth session of the Social Impact Academy. We learned that community feedback is a powerful tool for informing programmatic direction and achieving social impact. Gathering feedback can be done cheaply, but it should always be done systematically.

Session recording: http://www.meetingburner.com/b/globalgiving/view_recording?c=5IATZJ&h=f

Session summary:

David Bonbright, Chief Executive, Keystone Accountability

Keystone Accountability uses a practice called Constituent Voice.

Constituent Voice:

  • An assessment method which asks the people intended to benefit from social change what they think about plans, performance, and reports. It helps organizations become more effective within the communities they serve.
  • Constituent Voice requires building relationships. After you ask for feedback data you must report back to respondents.Let them know what feedback you received and what you plan to do.
  • These relationships lead to data improvement. The process of asking and discussing issues with the community lets people understand why they are being questioned. This encourages the community to give more useful feedback and data for future surveys.

Use this systemic approach to move from discussion to action:

  1. Design: You don’t have to ask 30 questions to get meaningful feedback, just ask one powerful one: “How likely would you be to recommend my company to a friend or colleague?”
  2. Collect: Perform micro-surveys, made up of 3-4 questions, continuously and record the data; it doesn’t cost much!
  3. Analyze: Once you have data, analyze it. There’s a lot you can learn using segmentation. Compare your responses with other organizations in your sector. The feedback commons will soon include a feature which allows comparison.
  4. Close the loop: This is the dialogue phase. Have a discussion with participants asking them meaningful clarifying questions.
  5. Course correct: You must follow through with improvements and communicate your actions to stakeholders. Repeat the process; it is cyclical!

Borrow techniques from the for-profit sector:

  • Keystone Accountability encourages use of the Net Promoter Score which asks the simple question: “On a scale from 0 – 10, How likely are you to recommend my organization to a colleague or friend?” Respondents are sorted into three categories: promoters (9-10) , passives (7-8), and detractors (0-6). The Net Promoter Score, however, is a quick, easy, and inexpensive way to collect rapid feedback from stakeholders. It makes it easy to benchmark your performance against other, similar organizations. And it allows you to compare past responses to current performance, in an effort to constantly improve.
  • Cisco is a tech company. Every time a Cisco employee has a meeting they ask: “Did you get what you wanted from this meeting?” The answer is then written down and reported. This changes how employees converse; they focus on the needs of the other person. Build a culture of responsiveness into your organization.

Sophie Sahaf, Vice President, Evaluation, LIFT

LIFT provides one-on-one support to help people lift themselves out of poverty.

How LIFT uses Constituent Voice

  1. It builds evidence for LIFT’s theory of change by asking poignant questions.Ex. LIFT’s theory is that it is important to include social, personal, and economic foundations to create better outcomes for communities. LIFT therefore asks questions that target feedback about social support and personal foundations. Questions might include: “Today, LIFT helped me with the goals and priorities that are most important to me”; and “I would recommend LIFT to a friend or relative.”
  2. It challenges assumptions of how members are benefiting from programs. Constituent Voice involves collecting and responding to member feedback which improves results.

How can you implement a survey with a small budget?

  • Keystone provides excellent free questions on their website. Borrow ideas from this and other organizations.
  • Use an iPad, email, telephone or computer to run the survey.
  • Keep surveys short, never longer than 5-8 questions.
  • Analyze data in house. 90% of LIFT’s data analysis is performed in excel. Much can be learned from simple statistics.
  • Continually modify the questionnaire. Change questions and learn as you go.

Make your feedback practical

  • Beware of asking vague questions. Surveys are more effective if respondents understand what you are asking.
  • You must have the leadership’s support to make community feedback data part of your programmatic implementation.
  • Time your surveys to internal planning cycles for annual performance goals or strategy planning.
  • Remember, not all feedback is meaningful. Analyze data for contradictions. Create focus groups to explain confusing results.

Social Impact Academy: How to Design Programs for Social Change

Social Impact Academy: How to Design Programs for Social Change

This week in the Social Impact Academy, we explored different approaches to implementing programs for social change. Sam Sternin, an international consultant, introduced us to Positive Deviance and Yennie Lee, the Impact Manager at IDEO.org shared about Human-Centered Design.

Session recording: http://www.meetingburner.com/b/globalgiving/view_recording?c=HC9ZWR&h=f

Session summary:

Introduction to Positive Deviance – Sam Sternin

Positive Deviance is one approach to behavior and institutional change.

Basic method of Positive Deviance:

  • Identify individuals whose behaviors have enabled them to achieve better outcomes than their peers despite having access to the same resources. If 90% of poor children are malnourished, that means 10% of them are nourished! Ask what that 10% is doing differently.
  • Design a program which encourages people to practice the deviant behaviors or strategies. Simply informing them does not work; only action creates change.
  • Monitor progress to reinforce positive behaviors.

Positive Deviance tries to tackle issues differently from traditional methods

Experts vs. communities

  • Experts say “you have a problem, we’ll fix it!” which is often not well-received or desired. Ideas from outside are difficult to transplant somewhere new which creates resistance to outside solutions.
  • Instead, make community members the experts! Ask communities what they care about and work to find solutions together.

Sustainable change vs. dependency

  • Outside solutions deepen the community’s dependency on outside actors.
  • Use solutions that already exist within the community. Home-grown solutions lead to more sustainable change.

Altering the social dynamics within communities

  • Identify community members who have already found a way to deviate from the norm within their social restrictions.
  • Rather than focusing on the educational ‘why’ behind new behaviors, have them start practicing the behavior immediately.

Let the community define what impact looks like

  • Community members should organize the program and monitor the progress with minimal outside facilitation.
  • Methodologies should be action oriented. Don’t wait to test something new!

Read examples of Positive Deviance in this introductory article.

Introduction to Human-Centered Design – Yennie Lee

Human-Centered Design is another problem-solving approach that can be useful to implement social change. It starts with people and ends with innovative solutions informed by those people’s needs. When you design from their perspective you have solutions that the people will embrace.

Good design is desirable (human), feasible (technology) and viable (business), but your emphasis should always be on the human element.

There are three phases of the design which should experience divergence and convergence in the process

  • 1. Inspiration: Open up to creative solutions for problems by challenging your assumptions on how things work. Use analogous inspiration- the deliberate practice of looking at other contexts that are related to the design challenge you are trying to solve.
    • For example, when IDEO was helping a U.S. hospital reconsider emergency room procedures, it visited a NASCAR pit team.
  • 2. Ideation: In this phase, make sense of what you have learned from the community and translate what you have learned into ideas. Always be inclusive of out-of-the-box ideas when brainstorming. Then take your inspiration and transform it into tangible prototypes that you can test in the community.
  • 3. Implementation: Take your prototype or program into the world and to see how the community reacts. Test multiple ideas for effectiveness. Get feedback by asking, “Are we meeting basic needs and expectations?”

Human Centered Design is meant to be flexible, there is a good chance that you are already practicing these steps!

Visit IDEO.org’s design kit for Human-Centered Design tools.

Social Impact Academy: How do you think about social impact?

Social Impact Academy: How do you think about social impact?

This week three panelists from GlobalGiving’s nonprofit partners shared how their organizations think about social impact in the second session of the Social Impact Academy. The discussion focused on how each organization defines, measures, and involves the community in social impact.

Session recording: http://www.meetingburner.com/b/globalgiving/view_recording?c=E45BG1&h=f

Session summary:

Joining us on the panel were:

Daniele Reisbig, Development Coordinator, The Nyaka AIDS Orphans Project
Nyaka AIDS Orphans Project began in 2001 as a response to the HIV/AIDS epidemic in Uganda. Initially the project focused on education for orphans and vulnerable children, but it now encompases a more holistic model incorporating education, health, and family support.

Daney Ramirez, Las Claras Director, Voces Vitales de Panama
Voces Vitales empowers female leaders through educational and psychological support. Las Claras is a project within Voces Vitales that specifically provides support for teen mothers.

Lisha McCormick, Chief Development Officer, Last Mile Health
Last Mile Health saves lives by bringing healthcare to some of the most remote communities in the world. It does so by working with community healthcare workers to successfully respond to communities’ needs.

What does it take for an organization to be social impact focused?

Daniele:

  • Bus Test: Does your organization pass the bus test? Do you have systems in place so that if, heaven forbid, a team member were hit by a bus, the organization could continue to carry out the project? Or, what if your organization disappeared? Would the community have been improved by your presence, or would it have developed a dependence on you?
  • Understand the ripple effect: Is your program empowering the community? Could you be negatively impacting the community by importing products from abroad that could be locally sourced?
  • Know the difference: Use qualitative and quantitative measurements to track outputs and outcomes.

Daney:

  • Engage the community: The impact of your project spreads from your immediate beneficiaries to their community. Likewise, the community directly affects the outcome of your project. For an organization to be social impact focused, it must be engaged in community activities and actively taking part in addressing the community’s issues, even when the issues are not directly related to the project. That relationship has facilitated Voces Vitales’s entrance into its community, which has provided it access to resources and knowledge that will shape the program’s success.

Lisha:

  • Make hard measurements: At Last Mile Health, that includes reduction of child mortality, reduction of maternal mortality, improved health screenings. If you can measure it, you can manage it. Measurement allows you to understand the problem, to demonstrate the milestones your organization has reached, and to demonstrate your effectiveness.
  • Stay true to your mission: At Last Mile Health, impact is about challenging the status quo and pushing back on cynicism when people say “that community is not worth it.”

How do you measure the success of your programs?

Daniele:

  • You don’t need large financial resources to measure success.
  • Use information that is naturally collected. For example, to evaluate our educational facilities we look at our students’ academic performance on national exams compared to the national average. For healthcare, we compare the number of services given from year to year and conduct follow up evaluations to assess if the interventions work, patients had to come back, and if kids growing and developing.

Daney:

  • Keep good records. Vital Voices uses a variety of integrated record-keeping processes to track and measure impact. For example, health care bills and attendance records help track the frequency of visits. Satisfaction surveys help measure and improve the quality of care to keep teen moms coming back.
  • Assess outcomes. In addition to tracking activity, Vital Voices also assesses impact by conducting psychological evaluations and tracking high school diploma completion rates, vocational training completion, and jobs obtained.

Lisha:

  • Establish a credible baseline. Last Mile Health discovered that government census data was off by 60% so we conducted our own baseline assessment.
  • Use data to identify service gaps. Once we found that child mortality was higher than expected we performed a rigorous assessment to find where the health system broke down, including community surveys. We found that we weren’t providing enough services in neonatal health which hindered us from achieving the desired health outcomes.
  • Use measurements to set goals & track progress. Using baseline data, Last Mile Health identified specific goals for improved health outcomes including a 33% reduction in child mortality, a 50% increase in antenatal services, and an end goal of 100% coverage by health workers for remote villages in Liberia. We hold ourselves accountable to impact and outcome indicators using  KPI (key performance indicators) assessments throughout the year, relying on both qualitative and quantitative data.

How is the community involved in determining your social impact?

Daniele:

  • Facilitate community ownership. The community is directly involved in our planning, programming, and evaluation. Community members take ownership over the organization through advisory committees and volunteering. We also involve self-organized and democratically led groups in productive dialogue. For example, our granny groups vote to determine how resources are used.

Daney:

  • Understand community issues that affect project success. The community that Vital Voices supports had serious security concerns. By demonstrating interest and concern for this community issue, Vital Voices earned trust and respect for their programs. The key is to inform the community of your project and get them involved. Without acceptance from the community your impact will be minimal.

Lisha:

  • Ensure that community member’s voices are part of critical conversations. We did this by setting up community health committees and by committing to work with traditional midwives. This ensures that those who are identified as health caregivers and leaders in communities are integrated into our programs. We also communicate our project evaluations to the community which closes the feedback loop and demonstrates the good we are doing.

What does social impact mean for your organization?

Daniele:

  • A source of credibility with the community and supporters; it’s how we show we are worthy to be there. It helps us advocate for change by showing how well we are doing in the community, and also on the fact that other kids in the country need help.

Daney:

  • Proves that we are using limited resources effectively. Social impact means being able to show results and gives credibility to your project. It also helps secure funding.

Lisha:

  • Mission-focused impact. We are looking to achieve closure in the access gap of everything from healthcare to jobs to education, a big task but something we are proud of. Initially we set our goals and measured social impact based on the number of people we helped. However, by the nature of our mission we operate in sparsely populated areas. To meet our goals we would have to go to areas with denser populations which was contrary to our mission. This taught our organization to pick impact milestones very carefully. We had to decide what we really wanted to strive towards. We chose to instead think about social impact as the number of communities we reach rather than individuals, although we measure both.