Categories
COVID-19

Was the stay-at-home order effective in downstate Illinois?

Introduction

Many people in downstate Illinois fairly questioned whether some of the social distancing measures that were imposed statewide, such as the stay-at-home order and the closing of nonessential businesses, were really necessary in downstate Illinois. Those restrictions were almost certainly important for Cook County and the collar counties around Cook County, because those counties were hit fairly early and hard with the pandemic. But many counties in downstate Illinois reported relatively few COVID-19 cases during the first few months of the pandemic, so the restrictions were perhaps less important here and perhaps the tradeoff that is often discussed between public health concerns and economic concerns could have been managed differently downstate.

I investigated this question by comparing the course of the pandemic in downstate Illinois with five of the eight states that did not enact statewide stay-at-home orders: Arkansas, Iowa, Nebraska, North Dakota, and South Dakota. I found that downstate Illinois had 41 percent fewer reported COVID-19 cases per capita on May 3 than those five states and I estimated that Illinois’ stay-at-home order resulted in 12 to 42 percent fewer cases per capita in downstate Illinois through May 3. Therefore, at least part of that 41 percent difference was likely due to the stay-at-home order. However, I also found that downstate Illinois had 46 percent more COVID-19 deaths per capita on May 10 than the five comparison states and that Illinois’ stay-at-home order did not result in fewer deaths per capita in downstate Illinois. It appears that the number of nursing facility deaths per capita, in particular, was much higher in downstate Illinois than in the comparison states.

Background

In addition to the five comparison states that I listed above, Utah and Wyoming also did not adopt stay-at-home orders, but their Western location makes them less similar to downstate Illinois than the other five states. Oklahoma also did not adopt a statewide stay-at-home order and I included Oklahoma as a comparison state in earlier versions of this post. However, because Oklahoma City, Tulsa, and several other cities in Oklahoma did adopt stay-at-home orders, representing about half of Oklahoma’s population, I decided to exclude Oklahoma as a comparison state in this update   (https://www.nytimes.com/interactive/2020/us/coronavirus-stay-at-home-order.html).  I used a broad definition of downstate Illinois for this comparison to include all counties in Illinois except Cook County and the five collar counties. Thus, downstate Illinois includes about 4.4 million people, or about one-third of Illinois’ total population. As Table 1 below shows, through May 28, Cook County and the five collar counties (DuPage, Kane, Lake, McHenry, and Will) were all among the 20 Illinois counties with the highest reported COVID-19 cases per capita and COVID-19-related deaths per capita. Overall, the rates of cases per capita and deaths per capita in Cook County and the collar counties were both about four-and-a-half times as high as the comparable rates in downstate Illinois.

Table 1. Illinois counties with the highest rates of COVID-19 cases per capita and COVID-19-related deaths per capita

The first full day of Illinois’s stay-at-home order was March 22 and the last full day was May 29, but some of the comparison states began to relax other social distancing restrictions on May 1. Therefore, I could not estimate the effects of Illinois’ stay-at-home order remaining in place after May 1 or I might have included the effects of relaxing those other restrictions. I did, however, want to estimate the effects of Illinois’ stay-at-home order over as long a period as possible. The CDC has advised that COVID-19 symptoms typically appear two to 14 days after exposure to the virus (https://www.cdc.gov/coronavirus/2019-ncov/symptoms-testing/symptoms.html). So, one would not expect an intervention to affect a state’s number of reported cases until at least two days after the intervention starts or stops. Therefore, I used case data through May 3 to estimate the effects of Illinois’ stay-at-home order in downstate Illinois. In February, the World Health Organization advised that the time from the onset of symptoms to death ranged from two to eight weeks for Chinese patients who had died (https://www.who.int/docs/default-source/coronaviruse/who-china-joint-mission-on-covid-19-final-report.pdf). More recent studies estimated that the time from the onset of symptoms to death is typically at least seven days, for a total of at least nine days from exposure to the virus to death. Therefore, I used death data through May 10 to estimate the effects of Illinois’ stay-at-home order in downstate Illinois. By the same reasoning and in order to obtain as long a pretreatment history as possible for each state, I considered Illinois’ stay-at-home order to start two days later than it actually did in my analyses of COVID-19 cases and I considered it to start nine days later than it actually did in my analyses of COVID-19 deaths.

Data

I considered how Illinois’ stay-at-home order affected three different outcomes in downstate Illinois as compared with the 5 comparison states: movement of people, COVID-19 cases, and COVID-19 deaths. For the movement of people, I used the community mobility data that Google has made available to researchers (https://www.google.com/covid19/mobility/), which tracks users’ visits to different categories of places, including residences, workplaces, retail and recreation, grocery stores and pharmacies, and parks and shows the changes in the number of those visits (or, in the case of residences, the length of those visits) from the pre-COVID-19 baseline. For the COVID-19 case and death data, I used the county-level data from the New York Times website (https://www.nytimes.com/article/coronavirus-county-data-us.html). I would have liked to also consider COVID-19 hospitalizations, but, unfortunately, there isn’t a comprehensive source of COVID-19 hospitalizations data, as there is for case and death data; some states, including Illinois, do not even report cumulative hospitalizations data. Table 2 shows how downstate Illinois compares to the five comparison states on the mobility, case, and death outcomes. The mobility data show the average percentage changes during the month of April as compared with the pre-COVID-19 baseline; the case and death data are updated through May 3 and May 10, respectively, for the reasons explained above.

Table 2. Community mobility and COVID-19 case and death data for downstate Illinois and the five comparison states

Table 2 suggests that Illinois’ stay-at-home order helped to reduce mobility in downstate Illinois to some extent. Surprisingly, downstate Illinoisans do not seem to have increased the amount of time that they spent at home more than residents of most of the five states without stay-at-home orders. However, downstate Illinoisans reduced their visits to workplaces more than residents of any of those five states and did not increase their visits to parks nearly as much as residents of any of those states. As compared with the weighted average of the five states without stay-at-home orders, downstate Illinoisans appear to have spent a similar amount of time at home, but visited retail and recreation places, workplaces, grocery stores and pharmacies, and parks less.

Before discussing the COVID-19 case and death data in Table 2, I should note concerns about the quality of that data. Most experts believe that the actual number of cases in the United States is much larger than the number of reported cases, with some estimating that the actual number of cases could be 5 to 20 times as high as the number of reported cases (https://www.businessinsider.com/real-number-of-coronavirus-cases-underreported-us-china-italy-2020-4). The death data are believed to be much more accurate, but are likely still undercounted, with the actual number of deaths being perhaps 50% to 150% higher than the number of reported deaths (https://www.nytimes.com/interactive/2020/04/28/us/coronavirus-death-toll-total.html). Because the underreporting of cases and deaths is probably not the same across states, it is difficult to draw strong conclusions from state-to-state comparisons.

With that qualification in mind, Table 2 suggests mixed results for Illinois’ stay-at-home order in downstate Illinois in terms of COVID-19 cases and deaths. Downstate Illinois reported fewer cases per capita through May 3 than four of the five comparison states and 41 percent fewer cases per capita than the weighted average of those states. However, downstate Illinois reported more deaths per capita through May 10 than four of the five comparison states and 46 percent more deaths per capita than the weighted average of those states.

The charts below show the growth curves for the total number of COVID-19 cases and deaths per 100,000 people in downstate Illinois and the five comparison states. The first full day of Illinois’ stay-at-home order was March 22, which is day 82 on the x axis of the charts. However, as discussed above, you would not expect Illinois’ stay-at-home order to impact the number of cases in downstate Illinois until at least day 84 and the number of deaths until at least day 91. I presented the data through day 160, which was June 8, but as I discussed above, I am focusing on the case data through May 3 (day 124) and on the death data through May 10 (day 131).

Figure 1. COVID-19 case growth curves in downstate Illinois and five comparison states
Figure 2. COVID-19 death growth curves in downstate Illinois and five comparison states

These two charts suggest the same, mixed conclusion as Table 2. Downstate Illinois’ COVID-19 case growth curve was noticeably flatter (better) than Iowa’s, Nebraska’s, and South Dakota’s curves from day 84 to day 124 and similar to the case growth curves for the other two comparison states. However, downstate Illinois’ COVID-19 death growth curve was noticeably steeper (worse) from day 91 to day 131 than all of the comparison states except for Iowa.

Methods

For a more careful comparison than the simple comparison to the population-weighted average of the five comparison states, I used four different analytical methods at three different geographic levels: states, counties, and health service areas ((HSAs, which are geographic groupings of counties that people travel among for routine medical care). At each geographic level, I first used linear regressions to identify the demographic variables that were significant predictors of COVID-19 cases or deaths at that level. I considered 23 different demographic variables in those regressions, which also included an indicator variable for whether the state, HSA, or county was subject to a stay-at-home order and a lagged value of the dependent variable prior to the modified starting date of Illinois’ stay-at-home order. For the reasons explained above, the dependent variable was either the number of reported cases per 100,000 people on May 3 or the number of reported deaths per 100,000 people on May 10. For the state-level regressions, I used robust standard errors to allow for heteroscedasticity. For the HSA- and county-level regressions, I used state-clustered standard errors to allow for correlation within states. Some HSAs include counties from different states; if an HSA included counties both in Illinois and in a comparison state, I divided the HSA into two areas for all of my analyses.

At the HSA and county levels, I also used nearest neighbor matching to match units in downstate Illinois and units in the comparison states on the predictor variables identified in the regressions described in the preceding paragraph, including a pretreatment value of the dependent variable. Nearest neighbor matching imputes the missing counterfactual value of the dependent variable for each unit using an average of the dependent variable values of similar units from the other group. I matched each HSA and county with the three nearest neighbors in the other group within a caliper of 10. This means that the distance between the unit being matched and each of the three units chosen as a match, as measured by the sum of the squared differences between their predictor variable values scaled by the inverse of the variance matrix of the predictor variables for the unit being matched, must be less than 10. Units that did not have three neighbors within a caliper of 10 in the other group were excluded from the analysis. I repeated all of the nearest neighbor matching analyses using a caliper of 5 and those results were very similar to the results reported below with a caliper of 10. I also used a bias adjustment in the nearest neighbor matching analyses, which adjusts the difference in the dependent variable values between matched units to account for differences in the values of their predictor variables. And I again used heteroscedasticity robust standard errors.

At the state and HSA levels, I also used synthetic control matching to match treated units with synthetic control units on the predictor variables identified in the regressions described above, including the lagged dependent variable. Synthetic control matching constructs a synthetic control unit for each treated unit by finding a weighted combination of the control units that matches the treated unit as closely as possible on the pretreatment averages of the predictor variables. An advantage to synthetic control matching is that a weighted combination of control units can often provide a better match for a treated unit than any individual control unit or even than an average of two or more control units. However, unlike with the nearest neighbor matching, I was not able to adjust for any bias in the synthetic control matching due to differences in the predictor variable averages. Also, synthetic control matching works best when the units have a significant history of pretreatment data available for the dependent variable, although including other covariates that are important predictors of the dependent variable as I did can offset that need to some extent. About 26 percent of the HSAs in downstate Illinois had no cases on the modified starting date of Illinois’ stay-at-home order and about 79 percent of the HSAs had no deaths on that date. So, many HSAs did not have a meaningful pretreatment dependent variable average for matching, which may have affected the results. I did not even conduct synthetic control matching at the county level, because about 98 percent of counties in downstate Illinois did not have any deaths on the modified starting date of Illinois’ stay-at-home order. So, the disadvantage of lacking meaningful pretreatment averages for matching seemed to outweigh any potential advantages of applying the method again at the county level.

Finally, at the state level, I also used a method called augmented synthetic control matching, which is similar to the standard synthetic control method, but adjusts the results for any remaining differences in the pretreatment values of the outcome variable and the other predictor variables between the treated unit and the synthetic control unit. For the reasons explained above, although the first full day of Illinois’ stay-at-home order was day March 22, I used March 24 as the treatment date for all of the synthetic control analyses of cases per capita and March 31 as the treatment date for all of the synthetic control analyses of deaths per capita.

Results

Table 3 summarizes the results of all eight analyses that I conducted. I included the regressions in the table, although I conducted the regressions mainly to identify predictor variables for the nearest neighbor matching and synthetic control matching; the results from the state-level regression, in particular, should not be given much weight because of the small sample size. All eight methods estimated that Illinois’ stay-at-home order reduced the number of COVID-19 cases per capita in downstate Illinois through May 3. That effect was statistically significant in only three of those analyses, but the fact that the effect was so consistently negative strongly suggests that Illinois’ stay-at-home order did reduce the number of cases in downstate Illinois by 12 to 42 percent. By contrast, seven of the eight methods estimated that the stay-at-home order actually increased the number of deaths per capita in downstate Illinois, but none of those effects was statistically significant. Therefore, Illinois’ stay-at-home order may have been at least partly responsible for the fact that downstate Illinois had 46% fewer cases than the comparison states on May 3, but other factors are probably mainly responsible for the fact that downstate Illinois had 41% more deaths than those states on May 10.

Table 3. Summary of analyses of effects of Illinois’ stay-at-home order on COVID-19 cases and deaths in downstate Illinois and the five comparison states

Discussion

Why might Illinois’ stay-at-home order have reduced the number of cases in downstate Illinois, but not the number of deaths? One possible explanation would be if downstate Illinois had tested less aggressively than the comparison states. In that case, downstate Illinois may simply have detected fewer less severe cases and may have underreported its cases to a greater extent than the comparison states. Two indicators of how aggressively a state is testing are the number of tests conducted per capita and the percentage of positive tests. States that tested less aggressively might have a lower number of tests per capita and a higher percentage of positive tests. Table 4 shows testing data for downstate Illinois and the 5 comparison states through May 3. Downstate Illinois conducted 5 percent fewer tests per capita than the comparison states, but had a 37 percent lower positive test rate than those states. So, the comparison is mixed, but it does not appear that downstate Illinois had tested significantly less aggressively than the comparison states through May 3.

Table 4. COVID-19 testing data for downstate Illinois and the five comparison states

A second possible explanation would be if the comparison states had a greater percentage of people that were at risk of serious complications from COVID-19 because of age or preexisting medical conditions. The Kaiser Family Foundation (https://www.kff.org/global-health-policy/issue-brief/how-many-adults-are-at-risk-of-serious-illness-if-infected-with-coronavirus/) calculated the percentage of at-risk adults in each state — people older than age 64 and other adults with certain preexisting conditions such as heart disease, chronic obstructive pulmonary disease, uncontrolled asthma, diabetes, or a body mass index greater than 40. I used their methods to calculate that percentage for downstate Illinois. Overall, about 26% of downstate Illinois’ population (about 35% of the adult population) is at higher risk due to age or preexisting conditions, which is similar to the percentages for Iowa, Nebraska, North Dakota, and South Dakota and less than the percentage for Arkansas. So, downstate Illinois’ population as a whole does not seem more susceptible to the virus than the populations in the other states.

A third possible explanation would be if a greater percentage of at-risk people actually became infected in downstate Illinois than in the comparison states, even though the overall populations in the two groups have similar at-risk percentages. In particular, nursing facility residents have comprised a large percentage of COVID-19-related deaths in many states. So, if downstate Illinois had a greater percentage of nursing facility deaths, whether because of inferior procedures or simply due to bad luck, that could help to explain why Illinois’ stay-at-home order did not reduce the number of deaths per capita, even though it reduced the number of cases per capita. I could not obtain the historical data through May 10, but through May 28, about 71 percent of the COVID-19-related deaths in downstate Illinois were nursing facility-related (https://www.dph.illinois.gov/covid19/long-term-care-facility-outbreaks-covid-19). Among the five comparison states, those percentages were 42 percent for Arkansas, 54 percent for Iowa, and 55 percent for Nebraska; North Dakota and South Dakota did not reported the number of nursing home-related deaths at that time (https://freopp.org/the-covid-19-nursing-home-crisis-by-the-numbers-3a47433c3f70). So, it does appear that downstate Illinois’ percentage of nursing facility-related deaths was higher than average. If that percentage was 20 points higher than the average for the five comparison states, that difference could explain all of the 46 percent difference in COVID-19 deaths per capita summarized in Table 2. In other words, without considering nursing facility-related deaths, downstate Illinois’ COVID-19 death rate through May 10 may have been similar to the rate in the comparison states. But the fact that the nursing facility-related death rate may have been twice as high in downstate Illinois as it was in the control group states is still concerning.

What should we conclude from this entire analysis? Illinois’ stay-at-home order seems to have had the desired effect on mobility in downstate Illinois, as its residents reported a smaller increase in visits to parks and larger decreases in visits to retail and recreation places and workplaces than residents of the five comparison states. The stay-at-home order may also have had the desired effect on reducing the number of COVID-19 cases in downstate Illinois, as it experienced 41 percent fewer cases per capita than the comparison states through May 3 and I estimated that the stay-at-home order reduced the number of cases per capita by 12 to 42 percent, so at least part of that 41 percent difference was likely due to the stay-at-home order. However, unexpectedly, downstate Illinois experienced 46 percent more COVID-19 deaths per capita through May 10 than the comparison states and that difference may have been especially pronounced for nursing facility-related deaths. Considering that deaths are the most important and most reliable indicator of the spread of the pandemic, it would be difficult to conclude that Illinois’ stay-at-home order was effective in significantly reducing the health impact of the pandemic in downstate Illinois, as compared with the five states without stay-at-home orders.

That conclusion does not suggest, of course, that no social distancing measures were needed in downstate Illinois. All five comparison states that did not enact stay-at-home orders did enact other social distancing measures to varying degrees. Of those five states, Iowa had the strictest social distancing measures, as most nonessential businesses were closed, restaurant dining rooms were closed, and gatherings of more than 10 people were prohibited. Indeed, Dr. Anthony Fauci said that Iowa’s restrictions were “functionally equivalent” to a stay-at-home order. However, some of those five states had much less strict measures, as many non-essential businesses remained open in Arkansas, North Dakota, and South Dakota. South Dakota, in particular, probably had the least strict measures among the 48 contiguous states, as restaurant dining rooms also remained open and there were no restrictions on gathering sizes. Therefore, downstate Illinois could perhaps have followed a less restrictive course, with no stay-at-home order and with some nonessential businesses allowed to remain open, without having a significant increase in COVID-19 deaths.

Categories
COVID-19

COVID-19 Case Projections for Illinois Counties

(Updated through July 29)

(I have stopped updating this post, because my model is not as useful in predicting the course of the pandemic in Illinois counties, now that different states have taken very different approaches to managing the pandemic, with widely varying results).

Many researchers have created models with projections for COVID-19 cases, hospitalizations, intensive care unit stays, and/or deaths. Some of the most widely cited are models created by researchers at the University of Washington Institute for Health Metrics and Evaluation (https://covid19.healthdata.org/united-states-of-america), the University of Pennsylvania Perelman School of Medicine (https://penn-chime.phl.io/), the Los Alamos National Laboratory (https://covid-19.bsvgateway.org/), and a diverse team of epidemiologists and data scientists under a project called Covid Act Now (https://covidactnow.org/). Many of those models are excellent and have been utilized by federal, state, and local policymakers to help guide their responses to the pandemic. I do, however, have three concerns about many of those models. First, most models require the researchers to make some important assumptions about parameters such as the number of people to whom each infected person can transmit the virus, the number of days that a person is infectious, and the effects of various types of social distancing measures on transmission. Second, many models are basing their assumptions on data from other countries, such as China, Italy, and Spain, but the pandemic may play out differently in the United States than it has in other countries. And third, most models are designed to produce national estimates, which is challenging, because the pandemic is at different stages in different parts of the country. Due partly to these issues and partly to more basic methodological differences, current models are often producing very different projections from one another (https://projects.fivethirtyeight.com/covid-forecasts/).

I have created a model that relies on the fact that some parts of the country are further along the growth curve of the pandemic to produce COVID-19 case projections for Illinois counties. I use data only from the 48 contiguous states (and the District of Columbia), with the expectation that those states provide a better comparison for Illinois than foreign countries do. And I make no assumptions about the virus or the effects of social distancing on transmission of the virus; instead, the model’s projections are based only on the actual data from the 48 states.

My model is a multilevel nonlinear growth model using county-level data from the New York Times on cases reported each day in each county; I did not include data from New York City, both because New York City seems to be a unique case in terms of the pandemic’s spread and because data from New York City are reported only at the city level in the New York Times data, and not at the county level. The fixed portion of my model estimates the shape of the growth curve in total cases per 100,000 people using a fractional polynomial. The random portion of my model includes both random intercepts and random slopes at the state level and at the county level; the random slopes at both levels are also fractional polynomials. So, the overall shape of the growth curve is determined by a combination of national, state, and county trends; the model has allowed state and county trends to have more influence on the shape of the growth curve over time. The model used to also adjust for several demographic variables, but those variables have become less important over time as more days of case data have become available, so each county’s growth curve is now determined entirely by national, state, and county case data over time.

Of course, my model still has many of the usual limitations associated with these models. First, the number of reported cases is still believed to underestimate the actual number of COVID-19 infections in most counties, so relying on those data is inherently problematic. Second, there have been many reported clusters of COVID-19 infections in places such as nursing homes and prisons, which are virtually impossible to predict. Finally, there is typically a lot of uncertainty in making projections and my model is no exception to that rule. I do not include confidence intervals for my point estimates below because those intervals are so wide that they are of little value. With those limitations in mind, the table below includes the actual number of total cases in each Illinois county through July 29 and the projected number of total cases for each county through September 30.

The following chart shows the actual number of new daily cases reported in Illinois through July 29 and the number of new daily cases projected by my model after that date through September 30. My model projects that the number of new daily cases in Illinois will continue to increase through at least the end of September.

How do the results of my model compare to the results from other models? The University of Washington model does not provide publicly-available county-level estimates, but that model projects that the number of new daily cases in Illinois will decrease slightly through the end of September, before beginning to increase again (https://covid19.healthdata.org/united-states-of-america/illinois). The University of Pennsylvania model does not provide publicly available estimates for Illinois. The Los Alamos model also projects that the number of new daily cases in Illinois will decrease slightly through at least September 9 (https://covid-19.bsvgateway.org/). The Covid Act Now model provides estimates of projected hospitalizations for both the state of Illinois and individual counties in Illinois and projects that the number of hospitalizations in Illinois will decrease slightly through the end of August (https://covidactnow.org/us/il). So, all of those other models disagree with my model and project that, despite the increase in daily cases over the last couple of weeks, that new daily cases will instead decrease over the next month. I hope that those projections are correct, but, unfortunately, my model’s projection of a continuing increase in cases seems more realistic at this point.

Focusing on the downstate Illinois area, following are charts showing the actual and projected number of cases for several different health service areas. As for the state of Illinois as a whole, the growth curves for all of these health service areas are projected to continue to increase through at least the end of September.

I will continue to update this model occasionally. If you have any questions about the model, please feel free to contact me at grein3@uis.edu.