Evidence Portal

As a practitioners or policy-makers aiming to increase the health and well-being of people in low and middle income countries, you often lack information to decide how you can do the most good.

In the 10-minute video below we explain why the existing sources of information are often not enough, and why we therefore have developed the Value for Money Evidence Portal:

 

Summary of the evidence portal

Problem: Existing sources of evidence aren’t enough

Let’s say you work for a non-profit in the area of Sexual and Reproductive Health & Rights (SRHR) and you have 1 million EUROs to spend next year. You want to use this money to have the biggest possible impact in your sector. How are you now going to decide which health outcomes you will target with which interventions?

A first step would be to turn to the existing evidence. But this evidence is scattered across a large amount of academic publications, policy reports and online platforms, and typically presented in inaccessible and technical terms. Moreover, only a very small number of sources provide cost-effectiveness estimates to tell you how much you would have to spend to achieve a certain effect. Including costs into our consideration is crucial: we might prefer an intervention that is low-cost but which only has a moderate effect over an intervention that has a large effect but which is very expensive.

Cost-effectiveness estimates, however, only tell us how much value you can expect from different interventions for one specific outcome. So they are informative when you need to decide amongst several interventions targeting one outcome. But these estimates aren’t very helpful when you have more than one possible outcome you could target. To keep it simple, imagine you had identified two highly cost-effective interventions – community support groups for reducing neonatal mortality and hygiene education combined with improving sanitation facilities for reducing diarrhea incidences. How are you going to decide how to spend your 1 million EUROs across these two interventions? Just by looking at cost-effectiveness estimates, you still wouldn’t be able to assess how much money you should allocate to each intervention to achieve the biggest possible impact.

To figure out how you can do the most good with your limited budget, you would need to include at least two other factors: How burdensome (i.e. disabling) is the adverse outcome for people who experience it? And how many people experience this adverse outcome (i.e. how prevalent is it)? You might, for example, prefer a modestly cost-effective intervention that prevents HIV infections to a highly cost-effective intervention that prevents diarrhea in an area where the burden and prevalence of HIV/AIDS is much higher than for diarrhea. Both HIV infections and diarrhea are undesirable outcomes, but as you face a limited budget, you might attach a different weighting to them. The existing sources of evidence do not allow you to do so and therefore constrain the impact you can have.

Solution: Evidence portal with value for money estimates across different outcomes

We combine four different elements to calculate value for money estimates:

  1. The average cost per beneficiary, e.g. 12 EUR per beneficiary
  2. The average expected effectiveness of the intervention per beneficiary, e.g. a reduction of 15% of beneficiaries’ likelihood of experiencing diarrhea
  3. The weighting of the burden of the adverse health outcome the intervention is trying to improve, obtained via DALY estimates (e.g. 1 case of neonatal mortality has a DALY of 69, whereas 1 case of diarrhea has a DALY of 0,38).
  4. The prevalence of the adverse health outcome the intervention is trying to improve (e.g. 40% of children at the age of 0-5 experience 1 case of diarrhea).

If cost indications were not provided by published impact evaluations, we included cost estimations provided by involved NGOs working in this area. Data on effectiveness were obtained from published impact evaluations. To estimate the relative burden of a health outcome (element nr 3 above), we used DALY (disability adjusted life years) values. DALYs give an indication of how much suffering an adverse health outcome provides in terms of disability during one’s life and years of life lost. This allows us to express the relative burden between, for example, mild back-aches to blindness, and express this difference in numbers. Data on prevalence of adverse health outcomes were extracted from sources of the WHO, IMHE (Institute for Health Metrics and Evaluation) and scientific publications.

The value for money estimates indicate the EUROs a decision maker needs to spend in order to avert 1 DALY through a specific intervention in a geographical region. For example, we have estimated that improving water quality at its source in Sub Saharan Africa for reducing diarrhea rates costs 1088 EUR per averted DALY, while providing health education to communities for reducing diarrhea rates costs 24 EUR per averted DALY. This means that for averting a similar amount of burden for a similar number of people, we would have to pay 1088 EUR for the former intervention, but only 24 EUR for the latter intervention. Put differently, providing health education to communities provides 45 times more value for money than improving water quality at its source. As we are using value for money estimates instead of cost-effectiveness, we can now also compare the value we would get by investing in different interventions across different outcomes. For example, an intervention consisting of women’s support groups and health systems strengthening for reducing neonatal mortality costs 115 Euro per DALY averted.

The evidence portal also provides information about the reliability (i.e. the strength of the evidence) of the value for money, cost, and effectiveness estimates. Moreover, it includes an indication of the sustainability of the intervention (i.e. the impact of the intervention after it stops). Lastly, users can further explore the estimates we provide at for the various interventions by looking at summaries of the underlying impact evaluations and values we used to calculate the prevalence, burden, costs, and effectiveness estimates.

 

Frequently asked questions

When will this evidence portal be made accessible?

We hope to make this portal open-access in the near future – please send us an e-mail (see below) if you want us to let you know when it’s launched.

Which outcomes are included in the evidence portal?

The portal currently includes outcomes for Sexual and Reproductive Health & Rights (SRHR) and Water, Sanitation & Hygiene (WASH). In the future we plan to expand this to other health outcomes as well as social outcomes. 

How did you decide which impact evaluations to include in the evidence portal?

The portal currently only includes data obtained from the most rigorous impact evaluations: randomized controlled trials. In the future we would like to explore how we could also include other (quasi)experimental designs.

What are the limitations of this evidence portal?

  1. The helpfulness of the evidence portal is tied to the quality and availability of published impact evaluations. The portal does not include interventions that haven’t been rigorously evaluated and published. This means that it currently doesn’t include enough ‘complex interventions’ (targeting several outcomes and/or consisting of multiple components, e.g. combining health education for pupils at schools with improved access to local health care and condoms to reduce HIV infection rates).
  2. The value for money estimates show to what extent interventions have been reported to ‘work’, but not how and why interventions work. We recommend that policy-makers and practitioners combine the findings of this evidence portal with process evaluations.   

Who is using this evidence portal?

Two health NGOs, Simavi and Max Foundation, are working with the evidence portal to optimize their organizational strategies. They are reflecting on the types of interventions with which their specific organizations are likely to have the most impact. A key focus here is additionality: these organizations are trying to increase their impact by implementing interventions that other players in the field wouldn’t have done anyway.

The two health NGOs are also using the evidence portal at a more tactical level in three ways. For one, using the portal has made them realize that there is very little evidence for some of the interventions they are engaged in, and they are therefore currently prioritizing which interventions they will rigorously evaluate to contribute to the existing evidence base. Second, the NGOs have dressed down their current M&E efforts at the outcome-impact level for interventions which are already strongly backed by existing evidence. Third, the NGOs are attempting to refrain from certain interventions due to the prevailing evidence of their ineffectiveness.

Who developed this evidence portal?

This evidence portal was developed under the lead of Erasmus University Rotterdam researchers Dr. Kellie Liket and Dr. Vera Schölmerich, assisted by Marjolijn Hekelaar and Yveta Melounová. The portal is supported by the Impact Center Erasmus, the two health NGOs Simavi and the Max Foundation, and the software company Matthat.

 

Feel free to get in touch with us for more information:

Dr. Kellie Liket, Impact Center Erasmus
kellieliket@gmail.com
LinkedIN

Dr. Vera Schölmerich, Erasmus University College
vera.schoelmerich@cantab.net
LinkedIN