This resource covers monitoring and evaluation (M&E) principles and practical ways of applying these to hygiene during infectious disease responses. We also provide some general principles for how to understand whether your project is having an impact on relevant preventative behaviours.

M&E allows for the assessment of the performance of a project. M&E should be thought of as a single process, utilizing data collected on an ongoing basis and at different time points. M&E data supports learning and accountability for all stakeholders, including funders, beneficiaries, implementers, and policy makers.

Monitoring is a continuous process of collecting data throughout the lifecycle of a project. It involves the collection, analysis, communication, and use of information about the project’s progress. Monitoring systems and procedures should provide the mechanisms for the right information to be provided to the right people at the right time, to help them make informed decisions about project progress. Monitoring data should highlight the strengths and weaknesses in project implementation and enable problems to be solved, performance to improve, success to be built on and projects to adapt to changing circumstances. Often routine monitoring data feeds into process evaluations as described below.

Evaluation refers to the systematic assessment of whether a project is achieving its stated goals and objectives as determined at the design stage and/or the extent to which the program has resulted in the anticipated outcomes and impact among the target population and if any unanticipated outcomes or impact have resulted. Please see this resource for detailed definitions of each of these. Evaluations often take place at baseline, midterm and endline. Evaluation can be distinguished from monitoring and regular review by the following characteristics:

One of the most useful tools for developing an M&E plan is to develop a theory for how your project is likely to be able to achieve its outputs and the associated outcomes and impacts. This is what is often called a ‘theory of change’. Developing a theory of change will allow you to establish appropriate M&E processes.

A theory of change describes how your project proposes to bring about a change in behavior or health outcomes by outlining a step-by-step series of causal events. When developing a theory of change, it is useful to use a ‘backwards mapping’ approach, which starts with the desired outcome and then works backwards, identifying the short- and medium-term actions and objectives required to achieve this. A theory of change is based on assumptions about what needs to take place and incorporates an understanding of how the context might support or hinder the success of the intervention.

There are many different terms used to describe the various components of a theory of change. Organizations may use different structures to understand the causal nature of a theory of change, including logframes and impact pathways.

We use the following terms and definitions in this resource:

The utilisation of inputs and the delivery of project activities leads to a cascade of events where outputs result in behavioural outcomes and health impacts. If we do not see the change we expected from a project, then we have either ‘theory failure’ (our theory of change was wrong) or ‘implementation failure’ (we did not deliver the project correctly). Process evaluation helps determine if there is implementation failure. M&E along the theory of change can help determine if there is theory failure.

A worked example of theory of change applied to a handwashing behaviour change project for COVID-19

Here we include an example of a simplified theory of change for handwashing with soap for the control of COVID-19, though the example could be applied to other infectious diseases with similar modes of transmission. The actual theory of change for any individual project may be more specific and should include indicators that are more measurable.

A well-articulated theory of change can be used to inform M&E activities. Monitoring is an ongoing process and should primarily focus on indicators related to activities and outputs. It may also include routine assessment of project outcomes. Evaluation focuses primarily on the later stages of the theory of change, assessing the achievement of outcomes and impacts. The components covered by evaluations will depend on the type of evaluation being done.

There are several types of evaluations that could be used in the development, delivery, and assessment of public health interventions. These are defined below:

Process evaluation: These document how a project is implemented and what was actually delivered compared to pre-defined plans. Process evaluations often assess the following:

Adequacy evaluation: These evaluations measure the extent to which a program or project has met predefined targets among the intended population.

These targets could be behavioral (for example: percent of the population practicing handwashing at key times), access targets, or health targets.

In an adequacy evaluation, success is measured against predefined targets which are set out at the start of project activities. Because adequacy evaluations do not compare projects against a control group, an adequacy evaluation cannot determine what would have happened if the project had not taken place (an important part of determining causation). As such, adequacy evaluations do not directly indicate if the project caused the measured changes in the intended population. However, if an adequacy evaluation measures indicators directly related to project activities and outputs and there are no other projects that are ongoing in that area, we can conclude that the changes are likely a result of the project.

Impact evaluation: The “gold standard” in public health, impact evaluations (which include randomized trials or quasi-experimental studies) determine if an intervention, project or program statistically impacted a key indicator of interest (e.g. health). This will often include the measurement of medium to longer-term outcomes and will examine cause and effect to understand if changes in behaviour or health can be attributed to your intervention. Impact evaluations require a control group, a set of units (e.g., households, communities, schools) that did not receive the intervention. These types of evaluations can be expensive and complicated. The use of impact evaluations may not be feasible or advisable in certain contexts.

For more information on adequacy and impact evaluations (plausibility and probability), we recommend reading this article which clearly explains not only what to measure for this type of evaluation, but also what can be claimed or demonstrated through each type of evaluation.

In the context of outbreaks, the dominant approaches for M&E typically include monitoring and process or adequacy evaluations. During outbreaks, it is less common for impact evaluations to be used. This is because impact evaluations are typically more costly (involving external staff and tighter controls on how data is collected), introduce ethical challenges (such as the use of control groups), and require a behavioural or health outcome measure (likely to be challenging to collect with quality during outbreaks). For more information on why health measures are not widely used during outbreaks, see this resource. Instead, we recommend that project implementers prioritise assessing the progress of interventions against predetermined behavioural and access targets related to the project’s outcomes.

Given the scale of some outbreaks, there may be many organisations in your area collecting M&E data. To avoid duplication and data collection fatigue among your populations, try to coordinate your M&E processes with other organisations working in the same area.

Note that whilst this section uses COVID-19 as an example, principles can be applied to other similar diseases. Tracking the incidence of COVID-19 cases and mortality is not recommended as part of programmatic impact evaluations. The reasons for this are described below.

During disease outbreaks, diagnostic testing is often not done systematically, which leads to us not having a good understanding of the total number of people, like we experienced with COVID-19. As testing is not done in the same way across countries, it makes it difficult to compare fatality rates in different populations regionally and globally. In addition to issues with defining the number of cases, there are also differences with how deaths involving COVID-19 are defined and issues of hidden deaths (people dying from COVID-19 who are never tested). Frequently this is a challenge during cholera and Ebola outbreaks, where there is both health facility AND community cases and deaths, making it difficult to understand the actual total number of cases and deaths. As a result, caution needs to be taken when interpreting the numbers of cases and case fatality rates during disease outbreaks, especially when trying to compare them across countries.

Given the unprecedented duration and severity of the pandemic, capacity in diagnostics, surveillance and reporting, treatment measures and uptake of preventative measures improved. In March 14 2020, the CFR varied from 0.22% in Germany to 6.81% in Italy and 3.74% globally. Meanwhile a year later after the first confirmed case of COVID-19, the CFR varied from 2.86% in Germany to 3.17% in Italy to 2.29% globally. Three years later since the pandemic started, we still have CFR’s of 1.19%, 1.89% and 1.91% being reported in India, Brazil and Iran respectively.

When designing measures to understand the health impact of your programme a common challenge is attributing any changes in incidence of cases and mortality to any specific program. This is difficult because:

It is therefore advisable to measure programmatic impact across a theory of change to see whether the program achieved intended consequences along the pathway that could lead to a reduction in COVID-19 transmission, cases and resulting deaths. It is also important that expectations about what a programme evaluation will and will not achieve are managed from the outset so no one is expecting health impact data and understands why this isn't useful.

Sometimes the results of M&E processes may be disappointing or not as positive as your organisation expected. It is often tempting to not share these findings with others in the same way that you may share positive findings. However, within the WASH sector there is renewed interest in learning constructively from failures. The Nakuru Accord encourages WASH practitioners to:

General M&E Resources:

Authors: Matt Freeman

Reviewers: Peter Winch, Katie Greenland, Karine Le Roch

Version date: 28.5.2020