Recently, the World Health Organization has exhorted countries to fight the Covid-19 pandemic with other interventions in addition to vaccines. But for countries to mount a comprehensive and effective response, more than exhortation is needed. Policymakers must understand the benefits and burdens associated with various policy options. They also have to be equipped to rigorously and systematically compare these benefits and burdens, both when evaluating individual policies and when determining which policies to include in a legislative or regulatory package.

Policymakers often use economic evaluations to weigh the costs and benefits of health policies. Some Covid-related policies and interventions are similarly amenable to assessment. Policies that have been adopted or discussed during the Covid-19 response include improving ventilation in indoor spaces; monitoring wastewater; increasing the availability of vaccines, therapies, testing, or face coverings; providing financial incentives for vaccination; requiring vaccination, testing, or face coverings; investing in the development of new vaccines, therapies, or tests; enforcing capacity restrictions for certain venues; ensuring isolation of people who test positive; providing financial support for people who must isolate or who experience economic disruption; closing certain spaces; restricting domestic or international travel; and issuing stay-at-home orders. The amount and quality of evidence available on the costs and benefits of these interventions vary. For example, the efficacy of Covid-19 vaccines has been extensively studied, and cost-effectiveness analyses have been conducted for vaccination.1 In contrast, improving ventilation is believed to help reduce Covid-19 transmission, but its effects are less well understood.

Furthermore, making Covid-related policy decisions requires considering not only trade-offs between health outcomes and the direct costs of interventions such as providing tests or vaccines, but additional dimensions related to economic activity, distributive justice, and individual liberty. Whether economic evaluations consider all societal effects or effects on only the health care system will influence the benefits and costs that are identified and how they are assessed. Creating an “impact inventory,” as recommended by the Second Panel on Cost-Effectiveness in Health and Medicine, is one way that analysts can be transparent about the costs and benefits that are considered in economic evaluations.2 Such transparency is critically important for supporting decision making.

Particularly for nonpharmaceutical and social-policy interventions, economic evaluation of Covid-related policies will often involve examining the combined effects of interacting policies, rather than considering policies in isolation. Such evaluations should describe whether and how nonlinear effects (whereby policies have different results in combination than they do independently) were modeled and in what direction assumptions about these effects might push the results.

Types of Comparative Evaluation for Covid-19 Policies and Interventions.

Along with reporting the benefits and costs that were quantified as part of an analysis, economic evaluations can include explanations regarding whether or how data related to various outcomes were aggregated. Disaggregated data could be most appropriately used to create dashboards of results that policymakers can consult to support decision making. In contrast, results based on aggregated outcomes (which combine separate outcomes into a single metric) should reflect the weight that society assigns to each component — for example, the extent to which society is willing to trade improved health for decreased spending or improved overall population health for the equitable distribution of health outcomes. The table describes types of economic evaluation, which vary in their methods of quantifying results and their approaches to aggregation.

In addition to having data on the magnitude of benefits and burdens, decision makers benefit from having information about relative certainty regarding the effects of various policies. Some interventions, such as Covid-19 vaccination, have been assessed in randomized trials that provide a reasonable degree of certainty regarding their effects, although real-world complications such as the arrival of new variants can introduce uncertainty. Other interventions, such as financial incentives for vaccination, have been analyzed primarily in observational trials. Still others are based on only biologic or other forms of scientific plausibility. For all interventions, there is some degree of uncertainty about long-term effects, given the relatively short course of the Covid-19 pandemic and the inherent difficulty involved in projecting likely outcomes with or without intervention.

Economic evaluation offers various options for addressing uncertainty. Economists can conduct value-of-information analyses to quantitatively assess whether the value of additional research into specific inputs, such as the effectiveness of vaccine-uptake strategies, is worth the costs of conducting such studies.3 Other types of uncertainty, such as uncertainty regarding the potential burdens associated with imposing longer isolation periods, are harder to quantify. In such cases, dashboards of disaggregated data can note uncertainties and highlight information — such as the likely direction of a policy’s effects — that may be important for early decision making. In this way, economic evaluation differs from null-hypothesis–based decision making. Economic evaluations consider both the probabilities of various outcomes affected by policy choices and their consequences and provide results in the form of expected values or a distribution of possible outcomes (including the probability of a worst-case scenario being realized, for example). Such evaluations can also clarify when results depend on factors that may change over time or differ by location, such as Covid-19 transmissibility or prevalence.

Policymakers don’t need to wait until economic evaluations are complete to act. Rather, they can make initial decisions without full information and revise policies later, depending on the outcomes of evaluations. This approach should be coupled with clear messaging explaining that initial policies are provisional and subject to change as new information arrives. Research has described the challenges associated with “disinvestment” from ineffective medical interventions, and similar issues may arise when public health policies are initially adopted (or rejected) and decisions are subsequently reversed. The expected costs of changing course should therefore be included in the analysis of any policy that involves deviating from the status quo.

Critics of this type of approach might argue that in the midst of a pandemic that is still killing thousands of people globally every day, we don’t have time to engage in economic evaluation — that we should do the best we can, without fully weighing the costs and benefits of the options under consideration. In contrast, we believe the severity of the pandemic makes the need for evaluation all the more urgent. Choosing optimal interventions is associated with a bigger payoff when risks are higher. Acknowledging trade-offs is a more transparent approach than analyzing only outcomes that can be measured — or worse, only outcomes that will reaffirm an investigator’s or a policymaker’s preexisting beliefs.

Decision makers, even when armed with empirical and model-based assessments, must also make normative decisions about how various outcomes should be weighed on the basis of their value to society and whether to explicitly consider factors related to the distribution of costs and benefits within a population. Some forms of evaluation, such as traditional cost-effectiveness and benefit–cost analyses, weigh all benefits and burdens equally, regardless of how they are distributed, and evaluate them on a single scale. Certain newer approaches, such as distributional cost-effectiveness analysis, weigh outcomes for certain beneficiaries (for example, members of marginalized communities) more heavily than outcomes for other beneficiaries.4 Other approaches, such as extended cost-effectiveness analysis, can be used to evaluate options relative to their performance on multiple objectives, such as health promotion and financial-risk protection.5

Although policymakers and investigators still struggle to quantify and compare the effects of various Covid-related interventions, we are steadily amassing data that could help inform choices. The pandemic’s medical, social, and economic harms have been immense, and they warrant a continuous policy response. All decision makers use some type of mental model to weigh the pros and cons of various policy options. Rigorous economic evaluation formalizes this process. Value judgments will still be required, but economic evaluation can make the decision-making process more systematic, comprehensive, and transparent.

Disclosure forms provided by the authors are available at NEJM.org.