This is definitely not the summer people had been planning on, thanks to COVID-19. People are understandably frustrated by months of lockdown, and there is growing pressure on governors and county officials to reopen.

In deciding how widely and quickly to relax restrictions, state and local leaders will rely heavily on White House guidelines as well as projections concerning the spread and mortality of COVID-19. But how accurate can we expect those projections to be?

The projections come from statistical models. Epidemiologists at Imperial College of London used their model to predict the coronavirus would cause as many as 500,000 deaths in Britain and over 2 million deaths in the U.S. by October. Their paper prompted the lockdown policies adopted by both countries.

>>> Failures of an Influential COVID-19 Model Used to Justify Lockdowns

This use of this work was disturbing on several counts. As with any model, the results produced hinge on the assumptions fed into it. In this case, the epidemiologists’ report of their finding didn’t clearly state the assumptions they made. More alarmingly, the lawmakers that decided to use this model to make policy decisions did not require the team to make their codes publicly available.

When the report came out, I repeatedly asked the authors for their models so I could examine them myself. They never responded. Recently, however, some aspects of their code have been released and have been shown to have numerous problems including a lack of reproducibility, undocumented code and frequent bugs.

Failing to get the Imperial model, my colleague Norbert Michel and I looked at another publicly available COVID-19 model grounded on decades of long-standing epidemiological work. We tested its sensitivity along three major assumptions: the specification of mortality rates within intensive care units (ICUs), the asymptomatic infection rate, and the level of communicability.

>>> The Challenges of Forecasting the Spread and Mortality of COVID-19

COVID-19 mortality rates in hospital ICUs are much higher for the elderly and chronically ill than for others. We found that, by using assumptions of mortality rates ranging from 5% to 30%, the model projected anywhere between 78,000 and 357,000 deaths by Aug. 1.

Assumptions about the level of asymptomatic infection also has a large impact on COVID modeling. When varying asymptomatic rates between 15% and 55%, the model’s projections of COVID deaths experienced by Aug. 1 ranged from 118,000 to 394,000.

Assumptions concerning the virus’s level of communicability also greatly impact estimates of disease spread. Popularized nine years ago in the movie “Contagion,” the basic reproduction number, also known as R0, quantifies the average number of people than an individual will infect with the virus. Assuming an R0 value of 1.5, the model predicted 44,000 dead by Aug. 1. When the R0 value was increased to 3.5, the death projection soared to 1.05 million.

Obviously, current death totals of COVID-19 now exceed our lower-end estimates, which were produced in April. Regardless, our modeling demonstrates that, under a variety of reasonable assumptions, COVID models generate a wide range of forecasts. Furthermore, we are still significantly below the high-end forecasts suggested by the Imperial College modeling team.

>>> Tracking the Trends in the COVID-19 Cases

As state governors move toward reopening their economies, they may continue to use models in their policy decisions. As they do so, however, it is important that the models used for guiding policy be made publicly available and their assumptions be clearly stated. The models must also be constantly updated with new data and tested for robustness.

The public should always be willing to question these models and their assumptions. Key questions to ask are: “What assumptions have been made?” “How legitimate are those assumptions?” and “What happens if you use reasonable alternatives?”

Without answers to these questions, the models should not be used in making policy decisions. After all, their projections can turn out to be just as bad as the Imperial College report that was used as the basis for locking down two countries.