One of the most useful tools for developing an M&E plan is to develop a theory for how your project is likely to be able to achieve its outputs and the associated outcomes and impacts. This is what is often called a ‘theory of change’. Developing a theory of change will allow you to establish appropriate monitoring and evaluation processes.
What is a theory of change?
A theory of change describes how your project proposes to bring about a change in behaviour or health outcomes by outlining a step-by-step series of causal events. When developing a theory of change it is useful to use a ‘backwards mapping’ approach which starts with the desired outcome and then works backwards identifying the short- and medium-term actions and objectives required to achieve this. A theory of change is based on assumptions about what needs to take place and incorporates an understanding of how the context might support or hinder the success of the intervention.
Source: Matthew Freeman
There are many different terms used to describe the various components of a theory of change. Organizations may use different structures to understand the causal nature of a theory of change, including logframes and impact pathways.
We use the following terms and definitions in this resource:
Inputs: The raw materials required (e.g. money, materials, technical expertise, training, relationships and personnel) by your project in order to deliver activities and achieve the outputs and objectives
Activities: The process or actions taken that will transform inputs and resources into the desired outputs.
Outputs: The direct results of the project activities. All outputs are things that can be achieved during the period of the grant and are linked to the objectives and goals.
Outcomes: Specific statements of the benefits that a project or intervention is designed to deliver. These should support the goal and be measurable, time-bound and project-specific. Many projects have more than one objective.
Impact: The long-term, large-scale challenge that your program will contribute to addressing.
The utilisation of inputs and the delivery of project activities leads to a cascade of events where outputs result in behavioural outcomes and health impacts. If we do not see the change we expected from a project, then we have either ‘theory failure’ (our theory of change was wrong) or ‘implementation failure’ (we did not deliver the project correctly). Process evaluation helps determine if there is implementation failure. M&E along the theory of change can help determine if there is theory failure.
A worked example of theory of change applied to a handwashing behaviour change project
Here we include an example of a simplified theory of change for handwashing with soap for the control of COVID-19. The actual theory of change for any individual project may be more specific and should include indicators that are more measurable.
Source: Matt Freeman
Monitoring and Evaluation and the Theory of Change
A well-articulated theory of change can be used to inform monitoring and evaluation activities. Monitoring is an ongoing process and should primarily focus on indicators related to activities and outputs. It may also include routine assessment of project outcomes. Evaluation focuses primarily on the later stages of the theory of change, assessing the achievement of outcomes and impacts. The components covered by evaluations will depend on the type of evaluation being done.
There are several types of evaluations that could be used in the development, delivery and assessment of interventions in the context of COVID-19. These are defined below:
Process evaluation: These document how a project is implemented and what was actually delivered compared to pre-defined plans. Process evaluations often assess the following:
Fidelity and quality - Extent to which a project was implemented as planned (e.g. did all of the steps outlined in the initial proposal or project manual take place).
Completeness - This refers to the number of project activities delivered as planned. For hygiene projects this could include the number of hygiene kits distributed, the number of households visited or the number of radio adverts aired. This is also sometimes called the ‘dose delivered’.
Exposure - This describes the extent and frequency with which the target population actively engages with the project activities. For hygiene projects, this could include the use and continued use of hygiene kits, attendance at training sessions or household visits, and the number of people hearing media messages. This is also sometimes called ‘dose received’. Exposure is different to completeness because it focuses on real engagement. For example for completeness, you may document that radio messages have been broadcast 50 times but when assessing exposure you may find that your target population didn’t hear these messages as the radio station chosen was not widely listened to.
Acceptability and satisfaction - This describe the extent to which participants and community members felt that the project addressed issues relevant to them and was delivered in a manner that was acceptable and appropriate for them.
Inclusiveness - This refers to the extent to which your project reached all of the people it intended to reach including vulnerable groups. For more information about making projects inclusive see these resources.
More detailed definitions and information about each of these process evaluation terms can be found here. On this website you will also find a range of other resources on process evaluations.
Adequacy evaluation: These evaluations measure the extent to which a program or project has met predefined targets among the intended population. These targets could be behavioural (for example: percent of the population practising handwashing at key times), access targets, or health targets. In an adequacy evaluation, success is measured against predefined targets which are set out at the start of project activities. Because adequacy evaluations do not compare projects against a control group, an adequacy evaluation cannot determine what would have happened if the project had not taken place (an important part of determining causation). As such adequacy evaluations do not directly indicate if the project caused the measured changes in the intended population. However, if an adequacy evaluation measures indicators directly related to project activities and outputs and there are no other projects are ongoing in that area, we can conclude that the changes are likely a result of the project.
Impact evaluation: The “gold standard” in public health, impact evaluations (which include randomised trials or quasi-experimental studies) determine if an intervention, project or program statistically impacted a key indicator of interest (e.g. health). This will often include the measurement of medium to longer-term outcomes and will examine cause and effect to understand if changes in behaviour or health can be attributed to your intervention. Impact evaluations require a control group, a set of units (e.g. households, communities, schools) that did not receive the intervention. These types of evaluations can be expensive and complicated. The use of impact evaluations may not be feasible or advisable in the context of COVID-19.
For more information on adequacy and impact evaluations (plausibility and probability), we recommend reading this article which clearly explains not only what to measure for this type of evaluation but also what can be claimed or demonstrated through each type of evaluation.
Source: Matt Freeman
Want to learn more about general principles for monitoring and evaluation of COVID-19 related hygiene projects?
Editors notes
Author: Matt Freeman, Sian White, Fiona Majorin
Reviewer: Peter Winch, Katie Greenland, Karine Le Roch
Last updated: 02.07.2020