Tracking the incidence of COVID-19 cases and mortality is not recommended as part of programmatic impact evaluations. The reasons for this are described below.
Limitations of data on COVID-19 cases and mortality
COVID-19 cases
COVID-19 cases can be either suspected, probable or confirmed through a laboratory test. Most of the national or global numbers of COVID-19 cases are of confirmed cases. However, the extent of testing varies across countries (e.g. 0.1 tests per thousand people in Indonesia to more than 100 tests per thousand people in Iceland), and data on testing is sometimes unavailable or incomplete. Aside from differences in testing strategies and protocols, there are other factors that might influence counts of cases including detection, definitions, reporting and lag times and these factors also differ across countries. In many countries testing is limited to those who present with symptoms, however evidence suggests that people can be asymptomatic carriers and can transmit COVID-19.
COVID-19 Mortality rate
There are a number of different mortality frequency measures. When it comes to COVID-19, we often refer to case fatality rates, this is defined as the proportion of persons with a particular condition (cases) who die from that condition. In order to measure case fatality rate, it is necessary to know the total number of COVID-19 cases and the number of deaths due to COVID-19 among those cases. However, since testing is not done systematically we do not have a good understanding of the total number of people with COVID-19. Importantly, as testing is not done in the same way across countries, it makes it difficult to compare fatality rates in different populations. In addition to issues with defining the number of cases, there are also differences with how deaths involving COVID-19 are defined and issues of hidden deaths (people dying from COVID-19 who are never tested).
As a result, some caution needs to be taken when interpreting the numbers of cases and case fatality rates, especially when trying to compare them across countries.
When designing measures to understand the health impact of your programme a common challenge is attributing any changes in incidence of cases and mortality to any specific program. This is difficult because:
At the moment, in any one setting, there are multiple actions being taken to combat COVID-19. These include actions by governments, businesses, non-government organisations and individuals or communities. It is likely that any changes in the epidemiological trend will occur as a consequence of the combination of these actions.
Community level preventative actions will not have a direct effect on COVID-19 death rates. Death rates are more closely linked to the capacity of healthcare facilities to manage severe cases and the proportion of vulnerable individuals in the population.
It is therefore advisable to measure programmatic impact across a theory of change to see whether the program achieved intended consequences along the pathway that could lead to a reduction in COVID-19 transmission, cases and resulting deaths. It is also important that expectations about what a programme evaluation will and will not achieve are managed from the outset so no one is expecting health impact data and understands why this isn't useful.
Want to learn more about general principles for monitoring and evaluation of COVID-19 related hygiene projects?
Editor notes
Author: Fiona Majorin
Reviewed by: Peter Winch, Katie Greenland, Karine Le Roch
Version Date: 28.5.2020