Sustained Focus on Accuracy
Since the release of our SEIR based localized COVID-19 planning model three weeks ago, a significant amount of information has emerged about the coronavirus and how transmission is impacted by social distancing. Over that time, the Qventus team has been regularly refining and updating our model to improve accuracy and usability.
As a reference point, our original model has consistently performed well for near-to-medium term projections. Examples of the performance of our original model are shown below, compared to another widely used forecast model from IHME. Even when predicting 7 days ahead of time, our census forecasts show a 2-3x better accuracy relative to the day-ahead IHME forecast when compared for 4 states (the only states reporting cumulative and current hospitalization).
Model Comparison: % Absolute Error in Hospital Census Predictions Across States
At the same time, traditional SEIR models may not fully incorporate the effects of the unprecedented social distancing measures that have been put in place across the world. In order to better reflect the effects of these social distancing dynamics, we have also updated the model with an option to “fit observed social distancing growth curve” — to base predictions on data from countries and states that are (a) further along in the pandemic curve, and (b) have implemented and maintained strong social distancing measures.
With these updates, we believe the model to be the best predictor of critical resource capacity requirements for hospitals across the country for scenarios with and without social distancing, especially with the continued ability for users to tune model parameters to fit hyperlocal observations.
Evolving Understanding of the Efficacy of Social Distancing
More data have become available from around the world, especially from major regions where the spread of COVID-19 appears to have peaked or plateaued. Traditional SEIR dynamics incorporate social distancing by flattening and elongating the curve, which can still imply very large peaks and a large proportion of the population ultimately becoming infected.
To date, major regions such as China and Italy appear to have already peaked or are nearing their peaks, and our teams have been analyzing the latest data from these areas to further inform our model.
New cases in these regions have decreased faster than traditional SEIR models have anticipated. This is likely because of the unprecedented level of social distancing that has been enacted around the world.
Therefore, to account for this powerful impact, we have developed a composite social distancing infection growth curve using data from regions further along in the pandemic surge: Hubei, Italy, Spain, Germany, United Kingdom, Switzerland, Belgium, and France, and select US States (New York, Florida, Illinois, Michigan, Ohio, Texas). Effectively, this allows the growth rate driving our model to be dynamic over time — under the assumption that social distancing continues — without compromising the underlying SEIR mechanics of our approach.
Growth rates were calculated using reported known cases in the region over 4-day windows. Regions were aligned at day 0 based on when a critical percentage of the local population became infected.
We averaged the regional growth rates together and weighted them accordingly:
- 55% – United States: most relevant to our predictions
- 35% – Non-China Countries: we believe their experiences based on actions taken will be more relevant to the United States than China
- 10% – Hubei: the stark decline in their curve is less relevant to the United States based on the severe lockdowns and quarantine practices which are not feasible in the US.
- These parameters were tuned via extensive backtesting against observed data across US states.
The option to fit these regional growth curves to your forecasts is available now and can be accessed in the model by toggling “Fit Observed Social Distancing Growth Curve.”
Note that the only place in the world where the growth rate has fully stabilized is still Hubei, which enacted the strongest social distancing measures known so far. Therefore, the long-term effects of lower levels of social distancing — e.g., the approaches taken in the United States — are still difficult to forecast.
The Path Forward
This update to the model will deliver significantly different results outside of the near term forecast window. For the 7-day forecast window, in states with sufficient data for backtesting, we have been able to reduce absolute error for census predictions by a substantial amount (detail below).
Qventus Model: Before and After Comparison of % Absolute Error in Census Predictions
While the revised model results are encouraging and demonstrate that social distancing measures are having a beneficial impact on the spread of the disease, we caution organizations not to take this observation as cause to prematurely remove social distancing measures. The model projections assume sustained social distancing measures. If these measures are removed, COVID-19 resource strains will be much more significant and more reflective of the original version of the model that uses more traditional SEIR epidemiological estimates.
We now recommend using this updated social distancing-based model for near and medium-term planning as long as social distancing measures are maintained, but continuing to leverage the original model for longer-term “worst-case scenario” planning, especially if social distancing measures are relaxed. We will also be adding additional options shortly to show the effects of turning on and off social distancing at different points in the future.
Going forward, our team will continue to refine and fit our model to keep pace with the evolving understanding of the virus. If you or your organization would like to collaborate, please reach out to covid19@qventus.com.
