Categories
COVID-19

Almost All K-12 Schools Should Be Open at the Beginning of 2021

(Updated on January 7, 2021)

I have written a couple of previous blog posts explaining that in-person K-12 instruction, at least on a part-time basis, did not seem to be contributing significantly to the spread of the current pandemic. Social scientists usually hesitate to reach definitive conclusions based on research studies, because we understand the limitations of those studies and that any particular study, or even a whole group of studies using a similar methodology, could suggest a conclusion that turns out to be incorrect. However, at this point, there is such a significant accumulation of evidence from so many different sources that children can attend school, at least on a part-time basis, without contributing significantly to the spread of the pandemic that it seems safe to conclude that almost all K-12 schools should be open at the beginning of 2021, at least with a hybrid instruction model.

I will review a timeline of that evidence in this blog post. First, though, it is important to make a distinction between the populations included in different studies. Some studies have focused only on the relationship between in-person instruction and COVID-19 cases in school-aged children, while other studies have considered the relationship between in-person instruction and COVID-19 cases in the entire community (usually, the county or state). Studying the relationship only with school-aged children can be preferable because the confounding effect of other social distancing measures is likely smaller than it is when we consider the entire community. However, limiting the population to school-aged children fails to consider possible effects on teachers and other school staff and the possibility that children could be asymptomatic spreaders of the virus and it also doesn’t consider indirect effects of in-person instruction on the spread of the pandemic – when children are back in school, parents may go out more and contribute more to the spread. Therefore, it seems important to consider the results from both types of studies and, in particular, to see if results differ between the types.

On May 14, a group of researchers from the University of Kentucky published an article in Health Affairs (https://www.healthaffairs.org/doi/full/10.1377/hlthaff.2020.00608), one of the leading health policy journals, analyzing the effects of various social distancing policies on the COVID-19 case growth rate across U.S. counties during the initial spread of the pandemic. Although they found that both shelter-in-place orders and restaurant and entertainment business closures significantly reduced the case growth rate, they found that school closures did not significantly affect the case growth rate.

On June 8, a large group of researchers from the University of California Berkeley published an article in Nature (https://www.nature.com/articles/s41586-020-2404-8), one of the leading journals across all scientific disciplines, analyzing the effects of various social distancing policies on the COVID-19 case growth rate across U.S. states and across localities in five other countries (China, France, Iran, Italy, and South Korea) during the initial spread of the pandemic. In the United States, they found that stay-at-home orders, closing businesses, work-from-home policies, and “other social distancing” measures all significantly reduced the case growth rate, but they found that school closures did not significantly affect the case growth rate.

On July 8, a group of researchers from the University of Pennsylvania published a study in Clinical Infectious Diseases (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7454446/pdf/ciaa923.pdf), a leading infectious diseases journal, analyzing the effect of emergency declarations and school closures on the COVID-19 adjusted mortality rate across U.S. states during the initial spread of the pandemic. They found that each day of delay in declaring an emergency was related to a 5% increase in the mortality rate and that each day of delay in closing schools was related to a 6% increase in the mortality rate. However, this study did not control for other social distancing measures and the authors acknowledged that “both emergency declarations and the timing of school closures may be a proxy for the degree to which a state began to officially and unofficially implement significant social distancing. . . . Thus, our results may reflect how quickly states responded to news about the size and severity of the spreading pandemic, with emergency declarations and school closures being among the first official nonpharmaceutical interventions, rather than protective effects specific to either intervention itself.” I agree with that interpretation and do not consider this study to show that closing schools reduced the mortality rate.

On September 23, researchers from Brown University and Qualtrics first released their National COVID-19 School Response Data Dashboard (https://statsiq.co1.qualtrics.com/public-dashboard/v0/dashboard/5f78e5d4de521a001036f78e#/dashboard/5f78e5d4de521a001036f78e?pageId=Page_c0595a5e-9e70-4df2-ab0c-14860e84d36a), which they have continued to update. Their dashboard initially included data from only about 550 schools, but now includes data from more than 9,000 schools. They have not published any reports analyzing the dashboard data. However, Emily Oster, the lead Brown University researcher on the project, has been featured in numerous news stories since then and, based on the dashboard data, she has consistently expressed the opinion that schools do not seem to be major spreaders of COVID-19 (https://www.theatlantic.com/ideas/archive/2020/10/schools-arent-superspreaders/616669/; https://www.washingtonpost.com/opinions/2020/11/20/covid-19-schools-data-reopening-safety/?arc404=true ).

On September 27, I published a blog post (http://blogs.uis.edu/garywreinbold/2020/09/27/effect-of-k-12-instruction-types-on-reported-covid-19-cases-and-deaths-in-illinois-counties/) analyzing the effects of different K-12 instruction types on COVID-19 cases and deaths in Illinois counties. These results have not yet been peer-reviewed, so they should be interpreted more cautiously than studies published in high-quality journals. I compared three groups of Illinois counties based on the instruction types that they used at the beginning of the school year: counties with a majority of students in districts with in-person instruction, counties with a majority of students in districts with hybrid instruction, and counties with a majority of students in districts with online-only instruction. I found that majority online-only instruction and majority hybrid instruction both significantly reduced the number of new cases in the county as compared with majority in-person instruction. However, there was not a significant difference between majority online-only instruction and majority hybrid instruction in the number of new cases and there was not a significant difference between any of the three groups of counties in the number of new deaths.

On October 1, the Swiss non-profit organization Insights for Education released a report (https://blobby.wsimg.com/go/104fc727-3bad-4ff5-944f-c281d3ceda7f/20201001_Covid19%20and%20Schools%20Six%20Month%20Report.pdf) analyzing the relationship between school reopenings and new COVID-19 cases across countries. Their report also was not peer-reviewed. Also, they considered only the unadjusted relationship between school reopenings and cases, without using statistical methods to control for other factors, which gives me even less confidence in their findings. However, they concluded that “there has not been any consistent relationship between school closure dates and the reported cases of infection in the population.”

On October 2, a group of Spanish researchers released a report (https://biocomsc.upc.edu/en/shared/20201002_report_136.pdf) analyzing the relationship between reopening schools and the COVID-19 case growth rate across Spanish autonomous communities (which are similar to states). Their report also has not yet been peer-reviewed and, like the Insights for Education report, it considers only the unadjusted relationship between school reopenings and cases. However, after considering the relationships between school reopenings and both cases within the entire population in each autonomous community and cases by age group in each autonomous community, they concluded “that the global incidence evolution suggests no significant effects of the reopening of schools, and that, in most cases, there is either absence of increase in cases of pediatric ages or a slight increase that is compatible with current diagnostic effort in the schools.”

On October 16, two British researchers published an article in Science (https://science.sciencemag.org/content/370/6514/286), another leading journal across all scientific disciplines. Based on their review of the evidence about COVID-19 and schools, they concluded that “existing evidence points to educational settings playing only a limited role in transmission when mitigation measures are in place.” As a result, they expressed the opinion that “school closures should be undertaken with trepidation given the indirect harms that they incur. Pandemic mitigation measures that affect children’s wellbeing should only happen if evidence exists that they help because there is plenty of evidence that they do harm.”

On November 6, the Illinois Department of Public Health first released contact tracing data (Contact Tracing | IDPH (illinois.gov)), which it has continued to update weekly. I explained in a November 14 blog post (http://blogs.uis.edu/garywreinbold/2020/11/14/what-can-school-boards-learn-from-idphs-contact-tracing-data/) that schools were in the lowest risk category in the initial data release, with only about 0.24% of potential exposures resulting in outbreaks. Overall, I observed that the contact tracing data were not that helpful in making decisions about school instruction types, but that nothing in those data disagreed with the general conclusion that K-12 schools have not been a major contributor to the spread of the pandemic.

On November 19, at the White House’s coronavirus briefing, Robert Redfield, the Director of the Centers for Disease Control and Prevention (CDC) explained: “The infections that we’ve identified in schools, when they’ve been evaluated, were not acquired in schools. They were actually acquired in the community and in the household. . . . The truth is for kids K-12, one of the safest places they can be from our perspective is to remain in school.” (Trump Opposes Lockdown in Virus Surge But Birx Urges Vigilance – Bloomberg).

Finally, on December 15, a group of researchers from the University of Mississippi, Mississippi State University, and the CDC published a report (Factors Associated with Positive SARS-CoV-2 Test Results in Outpatient Health Facilities and Emergency Departments Among Children and Adolescents Aged <18 Years — Mississippi, September–November 2020 (cdc.gov)) in Morbidity and Mortality Weekly Report, which is a well-respected CDC publication, although it is not peer-reviewed. The authors studied 397 children in Mississippi who had taken COVID-19 tests between September 1 and November 5, including 154 children with positive test results and 243 children with negative test results. The children with positive tests were much more likely to have had close contact with a person with COVID-19 and to have attended gatherings with persons outside their household in the two weeks prior to the test. However, in-person school or child-care attendance was not associated with a positive test result.

Therefore, a variety of approaches have been used to analyze the relationship between in-person K-12 instruction and the spread of the pandemic and virtually all of those approaches have concluded that children can attend school, at least on a part-time basis, without contributing significantly to the spread of the pandemic. Of course, there may still be unusual cases where particular districts or schools should not offer any in-person instruction, such as if the virus is spreading rapidly in the community, if hospital or intensive care unit capacity in the community is critically low or, if the school’s facilities don’t allow in-person instruction to be safely offered even to smaller classes of students. However, other than those exceptional cases, it seems safe to conclude that almost all K-12 schools should be open at the start of 2021, at least with a hybrid instruction model.

A few new resources have been published relating to this issue since my original post, so I will briefly address those sources:

In late December 2020, a group of researchers from the University of Washington and Michigan State University posted a working paper (https://caldercenter.org/sites/default/files/WP%20247-1220_updated_typo.pdf) analyzing the relationship between school instruction types and new COVID-19 case rates in Michigan and Washington during September, October and November. Again, this paper has not yet been peer reviewed, so it should be interpreted more cautiously than some of the studies discussed above that were published in leading journals. However, these researchers found that hybrid instruction did not have any effect on COVID-19 case rates in Michigan (for any counties, regardless of their prior case rates), that in-person instruction did not have any effect on case rates in Michigan in counties with low to moderate case rates (below the 95th percentile) and that hybrid or in-person instruction (considered together) did not have any effect on COVID-19 case rates in Washington in counties with low to moderate case rates (below the 75th percentile). In counties with high case rates (above the 95th percentile in Michigan and above the 75th percentile in Washington), their results were inconclusive, as some methods found that in-person instruction (in Michigan) or hybrid or in-person instruction (in Washington) resulted in higher case rates, while other methods did not. Overall, this study suggests that hybrid instruction, in particular, has not significantly increased case rates in most counties, except possibly in counties with high pre-existing case rates.

On December 30, the California Department of Public Health posted its own summary of the existing evidence of the effects of school reopenings on the pandemic (https://www.cdph.ca.gov/Programs/CID/DCDC/Pages/COVID-19/Safe-Schools-for-All-Plan-Science.aspx) as part of the Governor’s effort to persuade California elementary schools to reopen in February (https://www.politico.com/states/california/story/2020/12/30/newsom-pushes-california-school-reopening-plan-that-could-begin-in-february-1351652). The CDPH summary concludes: “Core mitigation strategies are necessary for safe and successful schooling. If those mitigation strategies are implemented as several layers of safety, elementary schools can be safe workplaces for teachers and other staff and safe learning environments for children.” Considering that California has been one of the worst-hit states over the past month and has almost no available intensive care unit capacity in most parts of the state, it is notable that even they are attempting to reopen schools.

On January 4, three researchers from Tulane University published a report (https://www.reachcentered.org/uploads/technicalreport/The-Effects-of-School-Reopenings-on-COVID-19-Hospitalizations-REACH-January-2021.pdf) examining the effects of school reopenings on COVID-19 hospitalizations across the United States. Again, this report has not yet been peer-reviewed and the authors took some novel approaches that will likely benefit from peer review, so their results should be interpreted carefully at this point. However, the authors found no effect of school reopenings on COVID-19 hospitalizations for counties with low to moderate hospitalization rates (below the 75th percentile) and their results were inconclusive for counties with high hospitalization rates, with one method showing a small positive effect of school reopenings on hospitalization rates for those counties and another method showing no effect. Overall, this study suggests that school reopenings have not significantly increased hospitalization rates, except possibly in counties with high pre-existing hospitalization rates.

These additional resources add further support to the conclusion that almost all K-12 schools should be open at the start of 2021, at least with a hybrid instruction model, except in exceptional circumstances like those discussed above.

Categories
COVID-19

What can school boards learn from IDPH’s contact tracing data?

The Illinois Department of Public Health (IDPH) has started to release COVID-19 contact tracing data (http://www.dph.illinois.gov/covid19/contact-tracing). As the IDPH explains, COVID-19 contact tracing is conducted mainly by local public health departments and involves identifying people with confirmed COVID-19 cases and their close contacts and asking those people to quarantine at home. A close contact is a person who was within 6 feet of a confirmed case for at least 15 minutes at any time starting 2 days before the person with the confirmed case first experienced symptoms or, if the person with the confirmed case is asymptomatic, starting 2 days before the person with the confirmed case first took a positive COVID-19 test. Of the 191,960 confirmed cases in Illinois from August 1 through October 24, health departments attempted to contact 69% of those people and actually interviewed 54% of those people, representing 102,864 people with confirmed cases who were interviewed. Those interviews identified 171,905 close contacts and health departments attempted to contact 70% of those close contacts and actually interviewed 57% of the close contacts, representing 97,314 close contacts who were interviewed (http://www.dph.illinois.gov/covid19/contact-tracing-data).

IDPH is releasing data on the locations of actual COVID-19 case outbreaks and potential COVID-19 case exposures that were identified through the contact tracing interviews. An outbreak means that five or more cases from different households are linked to that location within a 14-day period (http://www.dph.illinois.gov/covid19/outbreak-locations). An exposure means that a person with a confirmed case visited that location during the 14 days before the person with the confirmed case first experienced symptoms or, if the person with the confirmed case is asymptomatic, during the 14 days before the person with the confirmed case first took a positive COVID-19 test (http://www.dph.illinois.gov/covid19/location-exposure). In its initial release, IDPH provided data on outbreak locations for two different time periods – from July 1 to November 6, and from October 8 to November 6. IDPH provided data on exposure locations only for the period from October 8 to November 6. (IDPH did not explain why the contact tracing data were updated only through October 24, while the outbreak and exposure locations data were updated through November 6.)

In addition to the general data that identifies categories of locations with outbreaks and exposures, IDPH released more specific data for the outbreaks that were linked to schools and the potential exposures that occurred in schools. The outbreak data identify the school and the approximate number of cases involved in each outbreak (www.dph.illinois.gov/covid19/school-aged-metrics). The exposure data identify the number of potential exposures for each school (www.dph.illinois.gov/covid19/school-exposures).

Although it is certainly helpful that IDPH has started to release these contact tracing data, it is difficult for school boards to use these data to make decisions about school instruction types without additional analysis. I have already seen several news articles with statements that schools are the third largest exposure risk or that schools are the largest source of new cases. Both of those statements are misleading, as I will discuss below.

The potential exposure data are particularly difficult to use. To consider the potential exposure risk at a location, a person would like to know the probability of contracting the virus at that location if the person follows recommended safe practices. But IDPH’s exposure data tell us only the number of people who visited the location within 14 days of having symptoms or taking a positive test, which has two significant limitations.

First, the 14-day period used by IDPH is very conservative. Researchers believe that people become contagious only 2-3 days before they develop symptoms and that people who test positive but never develop symptoms are not likely to be contagious after 10 days (https://www.health.harvard.edu/diseases-and-conditions/if-youve-been-exposed-to-the-coronavirus). So, it seems very unlikely that someone who first developed symptoms or took a positive test 14 days after visiting a location would have exposed people at that location.

Second, even if a person is contagious and visits a location, the risk of another person contracting the virus from that person at that location obviously depends on many other factors, especially how closely and how long those two people interact with each other, if at all, and the safety practices that those two people observe. And those factors vary significantly by location. Combining the lists of exposure locations and outbreak locations helps to show the relative risk of each location, although this task is made a little more difficult by the fact that IDPH uses slightly different categories for the locations of exposures and outbreaks. However, I matched the two lists as closely as I could and the table below shows, for the 19 location types with the most potential exposures, the number of potential exposures, the number of outbreaks, and the percentage of potential exposures resulting in outbreaks over the 30-day period from October 8 to November 6.

We cannot make too fine of distinctions based on only one month of data. However, there appear to be three basic categories of risk in this table. Community events and mass gatherings seem to be the highest risk locations, with almost 2% of potential exposures leading to outbreaks. Places of worship and correctional facilities seem to be the next highest risk locations, with almost 1% of potential exposures leading to outbreaks. And all other locations, including schools, seem to be of lower risk, with no more than 0.5% of potential exposures leading to outbreaks. Most locations fall into this last category; with more data, we might be able to make further distinctions within that category. Therefore, although it is true that schools had the third most potential exposures during this period, schools seem to be in the lowest risk category in terms of the percentage of those potential exposures that resulted in actual outbreaks.

The outbreak data are also difficult to use and there is a risk that school boards will place too much importance on the relative size of each pie slice, without considering the overall size of the pie. It appears from the IDPH data that relatively few new COVID-19 cases are associated with outbreaks of five or more cases at a single location, other than long-term care facilities. The IDPH data counted 76 outbreaks from October 8 to November 6. However, those data did not include outbreaks at long-term care facilities, which are reported on a separate IDPH page (http://www.dph.illinois.gov/covid19/long-term-care-facility-outbreaks-covid-19). And it seems that those data also did not include outbreaks in the City of Chicago, which were reported in a separate document prepared by the Chicago Department of Public Health (https://www.chicago.gov/content/dam/city/sites/covid/CommunityOutbreaks/CDPH Congregate & Community Outbreak Responses 11062020_FINAL.pdf). IDPH did not report the number of cases that were included in those 76 outbreaks, but it seems likely that fewer than 1,000 of the new cases reported from October 8 to November 6 were associated with those 76 outbreaks. Over that 30-day period, more than 155,000 new cases were reported in Illinois; about 36,000 of those cases were from the City of Chicago, so about 119,000 new cases were reported in the rest of the state. Thus, it seems likely that less than 1 percent of the new cases in the rest of the state over that 30-day period were associated with the 76 outbreaks. So, the data on the locations of outbreaks are not that helpful, considering that they seem to include the sources for less than 1 percent of new cases. And while it is true that more outbreaks occurred at schools than at any other type of location (other than long-term care facilities) during this period, the number of new cases associated with those school outbreaks was still very low – likely fewer than 100 new cases, which represented less than 0.1 percent of the new cases in the state (other than Chicago) over that period. To know how significant schools are as a source of new cases, we would need source data for the 99% of new cases that were not associated with these 76 outbreaks.

The data on the specific schools with outbreaks are also not as helpful as they could be, both because such a small percentage of new cases are associated with outbreaks and because the data don’t show what type of instruction each school was using when the outbreak occurred. Of the nine schools that were listed as having outbreaks, it appears from their websites and from news sources that all nine of the schools were using primarily in-person instruction at the time of their outbreaks. But it would be helpful if IDPH would provide that information, so school boards could better evaluate the risks of the different instruction types.

In summary, none of the contact tracing data released by the IDPH are very helpful for school boards in making decisions about instruction types. But, in general, nothing in the contact tracing data that have been released so far disagrees with the general conclusion that experts are increasingly reaching, which is that K-12 schools have not been a major contributor to the spread of the pandemic. The clearest support for this conclusion in the IDPH contact tracing data was a chart that was released with those data, but that was not actually derived from the contact tracing data, which showed weekly new cases among different age groups of children and young adults in Illinois (http://www.dph.illinois.gov/covid19/school-aged-metrics?countyName=Illinois). That chart shows that there was a significant spike between August 24 and September 6 in the number of cases among people 18 to 22 years old; this spike is consistent with news reporting about case spikes among students at colleges after classes resumed in the fall. However, there were no similar spikes in the number of cases among the two K-12 school age groups. All of the case numbers do trend upwards starting in early October, but that pattern is consistent with the overall upward trend in case numbers since then. I have reproduced IDPH’s chart below for convenience.

For more information on the effect of K-12 school instruction types on new COVID-19 cases in Illinois, please see my other blog post (https://blogs.uis.edu/garywreinbold/2020/09/27/effect-of-k-12-instruction-types-on-reported-covid-19-cases-and-deaths-in-illinois-counties/) and the NPR news article that I link to at the end of that post.

Categories
COVID-19

Effect of K-12 instruction types on reported COVID-19 cases and deaths in Illinois counties

(Updated on October 21)

Summary

Few decisions made by state and local governments in response to the coronavirus pandemic have affected families as much as decisions about K-12 instruction types – whether to provide in-person instruction, online-only instruction, or a hybrid of in-person and online instruction. Decisions about instruction types this fall have varied widely across states, counties, and school districts, partly because of differences in COVID-19 case metrics and partly for other reasons, including political differences. Despite this variation, it is difficult to determine exactly how much differences in instruction types have contributed to the spread of the pandemic, mainly because the instruction types were not randomly assigned to schools or districts, because many schools and districts have changed instruction types over time in response to changing community conditions, and because other factors that affect the spread of the pandemic may also have changed over time.

I estimated the effect of different K-12 instruction types on the spread of the pandemic in Illinois counties by comparing the average number of new daily reported COVID-19 cases and deaths over a three-week “post-treatment” period from August 24 to September 13, with the average number of new daily reported cases and deaths over a three-week “pre-treatment” period from August 3 to August 23. I compared three groups of counties based on the instruction type that a majority of county students used to start the school year. I used a synthetic control method that matched each county in each group with a combination of counties from each other group that were as similar as possible to it.

My results suggest that in-person instruction contributed significantly more to increases in the number of reported cases than either hybrid instruction or online-only instruction: having a majority of county students in hybrid districts may have resulted in about an 18% to 30% reduction in the number of new cases over the period from August 24 to September 13, as compared with having a majority of county students in in-person districts; and having a majority of county students in online-only districts may have resulted in about a 29% to 45% reduction in the number of new cases over that period, as compared with having a majority of county students in in-person districts. However, my results suggest that there was not a significant difference between hybrid and online-only instruction in contributing to increases in the number of reported cases. None of the differences in instruction types appear to have contributed significantly to increases in the number of reported deaths.

Introduction

In Illinois, as in almost all other states, many K-12 students did not return to their classrooms this fall, at least not on a full-time basis. An Education Week chart (https://www.edweek.org/ew/section/multimedia/map-covid-19-schools-open-closed.html) shows that only four states required schools to provide at least some in-person instruction, while two states required schools to provide only online instruction. In the remaining forty-four states, including Illinois, the type of instruction being provided varied by district or school. The Illinois State Board of Education (ISBE), in particular, left decisions about the type of instruction largely up to the 852 school districts in Illinois, although it did require districts to provide online instruction to any students whose parents requested it (https://www.isbe.net/Documents/Messsage-07232020.pdf). As a result, the type of instruction being provided has varied by district, with some districts providing only online instruction, some providing mainly in-person instruction, and some providing mainly a hybrid of online and in-person instruction. In some districts, the type of instruction being provided has varied by grade, with younger students receiving in-person or hybrid instruction and older students receiving online instruction.

The ISBE surveyed school districts in July about their plans for instruction to start the school year and posted the results of that survey in a dashboard on its website (https://www.isbe.net/coronavirus). According to that dashboard, 31% of districts (with 1,238,000 students) planned to provide only online instruction, 28% of districts (with 158,000 students) planned to provide mainly in-person instruction, and 41% of districts (with 527,000 students) planned to provide mainly hybrid instruction. Most Chicago-area districts and some of the larger downstate districts planned to provide only online instruction and most of the larger downstate districts planned to provide mainly hybrid instruction; districts that planned to provide mainly in-person instruction were mostly smaller downstate districts.

Like many social distancing restrictions that state and local governments imposed in response to the pandemic, decisions by school districts to keep schools closed have been controversial. Parents in many communities have protested to return to in-person instruction (https://chicago.cbslocal.com/2020/09/08/hundreds-turn-out-in-west-suburbs-to-protest-remote-learning-in-schools/), while teachers in some communities protested to continue online-only instruction (https://wgntv.com/news/coronavirus/ctu-holds-protest-calls-for-all-remote-learning/). A Pew survey in August found an important partisan divide regarding school reopening, as 36% of people who are or lean Republican said that schools should provide only in-person instruction and only 13% of those people said that schools should provide only online instruction, while 41% of people who are or lean Democratic said that schools should provide only online instruction and only 6% of those people said that schools should provide only in-person instruction (https://www.pewresearch.org/fact-tank/2020/08/05/republicans-democrats-differ-over-factors-k-12-schools-should-consider-in-deciding-whether-to-reopen/).

An important question, of course, was whether students could return to classrooms in a way that was safe for students, teachers, other school employees, and their families. In July, the Centers for Disease Control and Prevention (CDC) recommended that schools be reopened this fall, based on its conclusions that COVID-19 poses low risks to children and that children are not likely to be major contributors to the spread of the virus (https://www.cdc.gov/coronavirus/2019-ncov/community/schools-childcare/reopening-schools.html). However, the CDC’s recommendations were issued after President Trump and Education Secretary DeVos had stated that schools should be reopened and had threatened to withhold federal funding from districts that didn’t reopen, which led some people to question the basis of the CDC’s recommendations (https://www.washingtonpost.com/education/2020/07/28/democratic-lawmakers-probing-whether-cdc-guidelines-reopening-schools-were-influenced-by-political-pressure/).

Because almost all states closed all of their public schools at about the same time this spring during the start of the pandemic, it was almost impossible to determine what effect those school closings had on the spread of the pandemic. To determine the effects of a policy, there needs to be some variation in that policy, either across geographic units such as states, counties, or districts or across time within the same geographic units. Some researchers attempted to estimate the effect of the school closures this spring on the spread of the pandemic by comparing changes in the spread within states or counties before and after the closures occurred, but because so many other social distancing restrictions were being imposed at the same time, it was very difficult to disentangle the effect of the school closures from the effects of those other measures. However, because there was variation in the instruction types that Illinois school districts used to start the school year this fall, with some districts providing only online instruction, some providing mainly in-person instruction, and some providing mainly hybrid instruction, we can start to estimate the effects of these different instruction types on the spread of the pandemic.

Data Used

For data on school district instruction types, I started with the data on the ISBE dashboard, which stated that it had been updated through September 21. However, I observed that some information on the dashboard was not accurate. For example, Springfield District 186 is listed as providing hybrid instruction on the dashboard. But District 186 changed its plans in mid-August, after it had submitted its survey to ISBE, and has been providing only online instruction since the beginning of the school year. Therefore, I updated the ISBE dashboard data for the 200 largest districts in Illinois and, to the extent they weren’t included among the 200 largest districts, for the largest district in each county, by searching district websites and news sources. Many districts seem to have changed their plans in August, after they had submitted their surveys to ISBE. Some other districts may have misdescribed their instruction type to ISBE, as they characterized instruction that is in-person for all students except students whose parents requested online instruction as hybrid, when it should be classified as in-person. In some cases, school districts have changed instruction types since the beginning of the school year, with some districts switching to online-only instruction in response to school or community outbreaks and other districts starting to open schools for in-person instruction as community metrics improved. I coded the districts based on the instruction type that they were using at the beginning of the school year. When a district had different instruction types for different grades, I coded the district with the least restrictive instruction type being used (in-person is the least restrictive type, then hybrid, then online-only).

Because COVID-19 case and death data are available only at the county level, not at the school district level, all of my analyses are at the county level. I first divided the counties into three groups: counties where a majority of the students attended districts with primarily in-person instruction, counties where a majority of the students attended districts with primarily hybrid instruction, and counties where a majority of the students attended districts with only online instruction. I excluded Cook County, because of its unique characteristics, and I excluded six counties with relatively large numbers of college students (Champaign, Coles, DeKalb, Jackson, McDonough, and McLean Counties), because the resumption of college classes contributed to case spikes in many college towns. Of the 95 remaining counties, 41 were majority in-person counties, 32 were majority hybrid counties, and 17 were majority online-only counties; 5 counties did not have a majority of students in any of the three categories. The table below shows the counties in each of these three groups and the percentage of students in each type of district for each county.

Analysis Methods and Results

To determine whether there was a relationship between these three groups of counties and the number of newly reported COVID-19 cases or deaths, I used a method called synthetic control matching to conduct a series of analyses, matching counties from each group with counties from each other group. I matched the counties on ten demographic variables that were important predictors either of the number of newly reported cases or deaths or of which group the county was in: median household income, poverty rate, population density, size (land area of the county), median age, the percentage of residents who are Hispanic, the percentage of residents who are black, the percentage of residents who are Native American, the percentage of residents who have attended at least some college, and the county international migration rate (which measures the net percentage of people moving into or out of the county to or from other countries); I also included a pretreatment average of the number of daily reported cases or deaths as a predictor variable. With two groups of units (treated units and control units), synthetic control matching constructs a synthetic control unit for each treated unit by finding a weighted combination of the control units that matches the treated unit as closely as possible on the pretreatment averages of the predictor variables. The advantage to synthetic control matching is that a weighted combination of control units can often provide a better match for a treated unit than any individual control unit or even than an average of two or more control units.

Many Illinois schools started a week or two later than usual this year, with most schools starting sometime between August 24 and September 2. Therefore, I used August 24 as the treatment date in my analyses. Although a few schools started earlier than August 24, it usually takes at least a few days for COVID-19 cases to be detected and reported. Therefore, I would not expect school reopenings to noticeably affect reported COVID-19 case numbers until at least August 24; I would not expect school reopenings to noticeably affect reported COVID-19 death numbers until at least a few more days after August 24.

I conducted three synthetic control analyses each for reported cases and reported deaths: an analysis that compared majority in-person counties with majority hybrid counties, an analysis that compared majority hybrid counties with majority online-only counties, and an analysis that compared majority in-person counties with majority online-only counties. The table below shows the results of those six synthetic control analyses. The numbers in the table show the estimated effect of being in the more restrictive group on the number of reported cases or deaths per 100,000 people; negative numbers indicate that the more restrictive instruction type resulted in fewer cases or deaths. The asterisks in the table show whether the effect was statistically significant: three asterisks indicate that the effect was significant at a p-value of .01, meaning that it was at least 99% likely that there was an effect; two asterisks indicate that the effect was significant at a p-value of .05, meaning that it was at least 95% likely that there was an effect, and one asterisk indicates that the effect was significant at a p-value of .10, meaning that it was at least 90% likely that there was an effect; estimated effects without any asterisks were not statistically significant.

*** p < .01; ** p < .05; * p < .10

The second column of the table shows that majority hybrid counties had significantly fewer cases per 100,000 people than their synthetic control units of majority in-person counties from August 29 to September 10. Similarly, the sixth column of the table shows that majority online-only counties had significantly fewer cases per 100,000 people than their synthetic control units of majority in-person counties from August 24 to September 14 (except for a few days near the end of that period, when the effects were not quite statistically significant). However, there was no significant effect on cases per 100,000 people for majority online-only counties as compared with majority hybrid counties (except for one day when the effect was barely statistically significant). And there were no significant effects on deaths per 100,000 people in any of the comparisons.

For the comparison of cases in majority in-person counties and majority hybrid counties, the effect followed a U-shaped pattern, with no effect at the beginning of the period, a statistically significant negative effect in the middle of the period, and no effect at the end of the period. This U-shaped effect was not unexpected. It is not surprising that the difference between in-person and hybrid instruction did not begin to affect the number of reported cases for several days. It is also not surprising that, after several days, majority hybrid counties had significantly fewer reported cases than their synthetic control units of majority in-person counties, because in-person instruction generally involves having almost all students at school at the same time, while hybrid instruction usually means having less than half of students at school at any particular time. Finally, it is not surprising that the effect disappeared toward the end of the period, because many in-person districts that experienced school or community outbreaks changed quickly to hybrid or online-only instruction and some hybrid districts with favorable community metrics changed to in-person instruction. So, there was a significant amount of crossover between treated and control counties by the end of the period. The comparison of cases in majority in-person counties and majority online-only counties also showed a U-shaped effect pattern, although even on the first day of the period, majority online-only counties already had significantly fewer reported cases than their synthetic control units of majority in-person counties.

It is also not surprising that there was not a significant effect on deaths in any of the comparisons. It is generally more difficult to find significant effects on COVID-19 deaths than it is on cases, because the timing from infection to death varies much more than the timing from infection to confirmation of a positive case. And in this case, the population of students, teachers, and other employees at K-12 schools are generally not in the high-risk groups for severe complications from the virus as some populations are, such as nursing facility residents. Although there are undoubtedly some students, teachers, and other school employees who are in high-risk groups or who have immediate family members in high-risk groups, many of those people likely chose not to participate in in-person classes in in-person or hybrid districts.

The most unexpected result was that there was not a significant effect on reported cases in the comparison of majority hybrid counties and majority online-only counties. That result suggests that hybrid instruction did not contribute significantly to the spread of the pandemic. As noted above, hybrid instruction generally involves having fewer than half of the students in each class attend school at any particular time, which may have allowed the students and teachers to maintain an adequate distance to prevent significant transmission of the virus. Of course, screening and contact tracing likely also contributed, by keeping potentially infected people out of the classrooms and quickly controlling any outbreaks that did occur through quarantines.

I also conducted analyses comparing the three groups of counties using other analysis methods, such as difference-in-differences, difference-in-differences with kernel propensity score weights (PSW), propensity score matching, and propensity score regression. To have common time periods for all of the analyses, I compared the results from a three-week posttreatment period from August 24 to September 13 with the results from a three-week pretreatment period from August 3 to August 23. The effect estimates from the other methods were generally smaller than the synthetic control estimates and were almost never statistically significant. However, the synthetic control results were more consistent to different model specifications than the results from the other methods, so I have greater confidence in the synthetic control results. The table below summarizes the results for reported cases from the preferred version of each of these methods; the propensity score-based methods were not able to produce an estimate for the comparison of majority in-person counties and majority online-only counties.  The results for reported deaths were again small and never statistically significant, so I did not include them in the table.

** p < .05; * p < .10

So, having a majority of county students in hybrid districts may have resulted in about 4 to 8 fewer new daily cases per 100,000 people over that three-week period from August 24 to September 13 (an 18% to 30% reduction in the number of new cases), as compared with having a majority of county students in in-person districts; and having a majority of county students in online-only districts may have resulted in about 6 to 12 fewer new daily cases per 100,000 people over that period (a 29% to 45% reduction in the number of new cases), as compared with having a majority of county students in in-person districts. There again was not a significant difference between majority hybrid districts and majority online-only districts, although the fact that all of the estimates were negative suggests that online-only instruction may have had some small additional advantage over hybrid instruction in terms of reducing the spread of the pandemic, with perhaps 1 to 4 fewer new daily cases per 100,000 people over that period for majority online-only districts.

Limitations

There are several factors that could affect the accuracy of my estimates. The reliability of reported COVID-19 case data has improved as testing capacity has improved, but a large percentage of infections are likely still going undetected and unreported. The ISBE data on instruction types were not completely accurate and, even though I updated them for the 200 largest districts in Illinois and some additional districts that are the largest districts in their counties, there were likely still inaccuracies in my data. Even if the data were accurate, there is a significant amount of spillover between the three groups of counties, as people regularly travel from county to county and some students and teachers may even attend or work at a school in a different county than the county in which they live. And, as discussed above, there has also been a lot of crossover among districts and probably also among these three groups of counties, as districts have changed instruction types in response to changing school or community conditions.

Although I controlled for many important demographic differences between counties, I did not control for differences at the district or school level, which could also have affected my results. In particular, there may be important differences in facilities or resources between districts that started the school year with hybrid instruction and districts that started with online-only instruction, such that the online-only districts would not have been able to offer hybrid instruction as safely as my results suggest that the hybrid districts did. Also, my sample sizes were relatively small, especially for the number of majority online-only counties, which may have prevented me from finding statistically significant effects, especially for the comparison of majority hybrid counties and majority online-only counties. Finally, there may be other important factors that changed from August to September that affected my results, such as if some counties tightened or relaxed other social distancing measures. In particular, if there were differences in how fall sports and other extracurricular activities (either school-based or nonschool) were conducted, that could have affected my results; it is quite possible that counties or districts that were less restrictive in school instruction types were also less restrictive in other activities.

Conclusion

Overall, though, it appears that in-person instruction contributed significantly more to increases in the number of reported cases than either hybrid instruction or online-only instruction and that there was not a significant difference between hybrid and online-only instruction in contributing to those increases. None of the differences in instruction types appear to have contributed significantly to increases in the number of reported deaths. It will continue to be very difficult to accurately estimate the effect of different K-12 instruction types on the spread of the pandemic and there are significant limitations to my research, as I discussed above. However, it is important to start attempting to estimate these effects, so that school districts can make informed decisions about their instruction types. Of course, those decisions also depend on many other factors. Districts need to consider the academic and social effects of the different instruction types on children; the risks associated with the virus for children, teachers, other school employees, and their families; the effects of different instruction types on parents’ work schedules; and the financial costs for schools to safely reopen buildings to students. Research on any of these issues can help to inform those decisions.

It is important, however, that districts do not base their decisions solely on the current situations in their communities; they should also consider how their decisions will affect those situations. The fact that the community may have a higher number of cases or a higher positivity rate, in general, than a predetermined target does not necessarily mean that school must be conducted only online. The results from Illinois counties that I discussed above suggest that hybrid instruction may not have significantly contributed to additional increases in the number of new cases in the community, as compared with online-only instruction. A recent study from Spain and anecdotal evidence from Utah and other states have also indicated that school reopenings may not have contributed significantly to the spread of the pandemic (https://www.nprillinois.org/post/were-risks-reopening-schools-exaggerated). My results do not go that far, as they do suggest that full in-person instruction may have contributed to increases in reported cases in Illinois counties.

If you have any questions about this research, please contact me at grein3@uis.edu.

Categories
COVID-19

Was the stay-at-home order effective in downstate Illinois?

Introduction

Many people in downstate Illinois fairly questioned whether some of the social distancing measures that were imposed statewide, such as the stay-at-home order and the closing of nonessential businesses, were really necessary in downstate Illinois. Those restrictions were almost certainly important for Cook County and the collar counties around Cook County, because those counties were hit fairly early and hard with the pandemic. But many counties in downstate Illinois reported relatively few COVID-19 cases during the first few months of the pandemic, so the restrictions were perhaps less important here and perhaps the tradeoff that is often discussed between public health concerns and economic concerns could have been managed differently downstate.

I investigated this question by comparing the course of the pandemic in downstate Illinois with five of the eight states that did not enact statewide stay-at-home orders: Arkansas, Iowa, Nebraska, North Dakota, and South Dakota. I found that downstate Illinois had 41 percent fewer reported COVID-19 cases per capita on May 3 than those five states and I estimated that Illinois’ stay-at-home order resulted in 12 to 42 percent fewer cases per capita in downstate Illinois through May 3. Therefore, at least part of that 41 percent difference was likely due to the stay-at-home order. However, I also found that downstate Illinois had 46 percent more COVID-19 deaths per capita on May 10 than the five comparison states and that Illinois’ stay-at-home order did not result in fewer deaths per capita in downstate Illinois. It appears that the number of nursing facility deaths per capita, in particular, was much higher in downstate Illinois than in the comparison states.

Background

In addition to the five comparison states that I listed above, Utah and Wyoming also did not adopt stay-at-home orders, but their Western location makes them less similar to downstate Illinois than the other five states. Oklahoma also did not adopt a statewide stay-at-home order and I included Oklahoma as a comparison state in earlier versions of this post. However, because Oklahoma City, Tulsa, and several other cities in Oklahoma did adopt stay-at-home orders, representing about half of Oklahoma’s population, I decided to exclude Oklahoma as a comparison state in this update   (https://www.nytimes.com/interactive/2020/us/coronavirus-stay-at-home-order.html).  I used a broad definition of downstate Illinois for this comparison to include all counties in Illinois except Cook County and the five collar counties. Thus, downstate Illinois includes about 4.4 million people, or about one-third of Illinois’ total population. As Table 1 below shows, through May 28, Cook County and the five collar counties (DuPage, Kane, Lake, McHenry, and Will) were all among the 20 Illinois counties with the highest reported COVID-19 cases per capita and COVID-19-related deaths per capita. Overall, the rates of cases per capita and deaths per capita in Cook County and the collar counties were both about four-and-a-half times as high as the comparable rates in downstate Illinois.

Table 1. Illinois counties with the highest rates of COVID-19 cases per capita and COVID-19-related deaths per capita

The first full day of Illinois’s stay-at-home order was March 22 and the last full day was May 29, but some of the comparison states began to relax other social distancing restrictions on May 1. Therefore, I could not estimate the effects of Illinois’ stay-at-home order remaining in place after May 1 or I might have included the effects of relaxing those other restrictions. I did, however, want to estimate the effects of Illinois’ stay-at-home order over as long a period as possible. The CDC has advised that COVID-19 symptoms typically appear two to 14 days after exposure to the virus (https://www.cdc.gov/coronavirus/2019-ncov/symptoms-testing/symptoms.html). So, one would not expect an intervention to affect a state’s number of reported cases until at least two days after the intervention starts or stops. Therefore, I used case data through May 3 to estimate the effects of Illinois’ stay-at-home order in downstate Illinois. In February, the World Health Organization advised that the time from the onset of symptoms to death ranged from two to eight weeks for Chinese patients who had died (https://www.who.int/docs/default-source/coronaviruse/who-china-joint-mission-on-covid-19-final-report.pdf). More recent studies estimated that the time from the onset of symptoms to death is typically at least seven days, for a total of at least nine days from exposure to the virus to death. Therefore, I used death data through May 10 to estimate the effects of Illinois’ stay-at-home order in downstate Illinois. By the same reasoning and in order to obtain as long a pretreatment history as possible for each state, I considered Illinois’ stay-at-home order to start two days later than it actually did in my analyses of COVID-19 cases and I considered it to start nine days later than it actually did in my analyses of COVID-19 deaths.

Data

I considered how Illinois’ stay-at-home order affected three different outcomes in downstate Illinois as compared with the 5 comparison states: movement of people, COVID-19 cases, and COVID-19 deaths. For the movement of people, I used the community mobility data that Google has made available to researchers (https://www.google.com/covid19/mobility/), which tracks users’ visits to different categories of places, including residences, workplaces, retail and recreation, grocery stores and pharmacies, and parks and shows the changes in the number of those visits (or, in the case of residences, the length of those visits) from the pre-COVID-19 baseline. For the COVID-19 case and death data, I used the county-level data from the New York Times website (https://www.nytimes.com/article/coronavirus-county-data-us.html). I would have liked to also consider COVID-19 hospitalizations, but, unfortunately, there isn’t a comprehensive source of COVID-19 hospitalizations data, as there is for case and death data; some states, including Illinois, do not even report cumulative hospitalizations data. Table 2 shows how downstate Illinois compares to the five comparison states on the mobility, case, and death outcomes. The mobility data show the average percentage changes during the month of April as compared with the pre-COVID-19 baseline; the case and death data are updated through May 3 and May 10, respectively, for the reasons explained above.

Table 2. Community mobility and COVID-19 case and death data for downstate Illinois and the five comparison states

Table 2 suggests that Illinois’ stay-at-home order helped to reduce mobility in downstate Illinois to some extent. Surprisingly, downstate Illinoisans do not seem to have increased the amount of time that they spent at home more than residents of most of the five states without stay-at-home orders. However, downstate Illinoisans reduced their visits to workplaces more than residents of any of those five states and did not increase their visits to parks nearly as much as residents of any of those states. As compared with the weighted average of the five states without stay-at-home orders, downstate Illinoisans appear to have spent a similar amount of time at home, but visited retail and recreation places, workplaces, grocery stores and pharmacies, and parks less.

Before discussing the COVID-19 case and death data in Table 2, I should note concerns about the quality of that data. Most experts believe that the actual number of cases in the United States is much larger than the number of reported cases, with some estimating that the actual number of cases could be 5 to 20 times as high as the number of reported cases (https://www.businessinsider.com/real-number-of-coronavirus-cases-underreported-us-china-italy-2020-4). The death data are believed to be much more accurate, but are likely still undercounted, with the actual number of deaths being perhaps 50% to 150% higher than the number of reported deaths (https://www.nytimes.com/interactive/2020/04/28/us/coronavirus-death-toll-total.html). Because the underreporting of cases and deaths is probably not the same across states, it is difficult to draw strong conclusions from state-to-state comparisons.

With that qualification in mind, Table 2 suggests mixed results for Illinois’ stay-at-home order in downstate Illinois in terms of COVID-19 cases and deaths. Downstate Illinois reported fewer cases per capita through May 3 than four of the five comparison states and 41 percent fewer cases per capita than the weighted average of those states. However, downstate Illinois reported more deaths per capita through May 10 than four of the five comparison states and 46 percent more deaths per capita than the weighted average of those states.

The charts below show the growth curves for the total number of COVID-19 cases and deaths per 100,000 people in downstate Illinois and the five comparison states. The first full day of Illinois’ stay-at-home order was March 22, which is day 82 on the x axis of the charts. However, as discussed above, you would not expect Illinois’ stay-at-home order to impact the number of cases in downstate Illinois until at least day 84 and the number of deaths until at least day 91. I presented the data through day 160, which was June 8, but as I discussed above, I am focusing on the case data through May 3 (day 124) and on the death data through May 10 (day 131).

Figure 1. COVID-19 case growth curves in downstate Illinois and five comparison states
Figure 2. COVID-19 death growth curves in downstate Illinois and five comparison states

These two charts suggest the same, mixed conclusion as Table 2. Downstate Illinois’ COVID-19 case growth curve was noticeably flatter (better) than Iowa’s, Nebraska’s, and South Dakota’s curves from day 84 to day 124 and similar to the case growth curves for the other two comparison states. However, downstate Illinois’ COVID-19 death growth curve was noticeably steeper (worse) from day 91 to day 131 than all of the comparison states except for Iowa.

Methods

For a more careful comparison than the simple comparison to the population-weighted average of the five comparison states, I used four different analytical methods at three different geographic levels: states, counties, and health service areas ((HSAs, which are geographic groupings of counties that people travel among for routine medical care). At each geographic level, I first used linear regressions to identify the demographic variables that were significant predictors of COVID-19 cases or deaths at that level. I considered 23 different demographic variables in those regressions, which also included an indicator variable for whether the state, HSA, or county was subject to a stay-at-home order and a lagged value of the dependent variable prior to the modified starting date of Illinois’ stay-at-home order. For the reasons explained above, the dependent variable was either the number of reported cases per 100,000 people on May 3 or the number of reported deaths per 100,000 people on May 10. For the state-level regressions, I used robust standard errors to allow for heteroscedasticity. For the HSA- and county-level regressions, I used state-clustered standard errors to allow for correlation within states. Some HSAs include counties from different states; if an HSA included counties both in Illinois and in a comparison state, I divided the HSA into two areas for all of my analyses.

At the HSA and county levels, I also used nearest neighbor matching to match units in downstate Illinois and units in the comparison states on the predictor variables identified in the regressions described in the preceding paragraph, including a pretreatment value of the dependent variable. Nearest neighbor matching imputes the missing counterfactual value of the dependent variable for each unit using an average of the dependent variable values of similar units from the other group. I matched each HSA and county with the three nearest neighbors in the other group within a caliper of 10. This means that the distance between the unit being matched and each of the three units chosen as a match, as measured by the sum of the squared differences between their predictor variable values scaled by the inverse of the variance matrix of the predictor variables for the unit being matched, must be less than 10. Units that did not have three neighbors within a caliper of 10 in the other group were excluded from the analysis. I repeated all of the nearest neighbor matching analyses using a caliper of 5 and those results were very similar to the results reported below with a caliper of 10. I also used a bias adjustment in the nearest neighbor matching analyses, which adjusts the difference in the dependent variable values between matched units to account for differences in the values of their predictor variables. And I again used heteroscedasticity robust standard errors.

At the state and HSA levels, I also used synthetic control matching to match treated units with synthetic control units on the predictor variables identified in the regressions described above, including the lagged dependent variable. Synthetic control matching constructs a synthetic control unit for each treated unit by finding a weighted combination of the control units that matches the treated unit as closely as possible on the pretreatment averages of the predictor variables. An advantage to synthetic control matching is that a weighted combination of control units can often provide a better match for a treated unit than any individual control unit or even than an average of two or more control units. However, unlike with the nearest neighbor matching, I was not able to adjust for any bias in the synthetic control matching due to differences in the predictor variable averages. Also, synthetic control matching works best when the units have a significant history of pretreatment data available for the dependent variable, although including other covariates that are important predictors of the dependent variable as I did can offset that need to some extent. About 26 percent of the HSAs in downstate Illinois had no cases on the modified starting date of Illinois’ stay-at-home order and about 79 percent of the HSAs had no deaths on that date. So, many HSAs did not have a meaningful pretreatment dependent variable average for matching, which may have affected the results. I did not even conduct synthetic control matching at the county level, because about 98 percent of counties in downstate Illinois did not have any deaths on the modified starting date of Illinois’ stay-at-home order. So, the disadvantage of lacking meaningful pretreatment averages for matching seemed to outweigh any potential advantages of applying the method again at the county level.

Finally, at the state level, I also used a method called augmented synthetic control matching, which is similar to the standard synthetic control method, but adjusts the results for any remaining differences in the pretreatment values of the outcome variable and the other predictor variables between the treated unit and the synthetic control unit. For the reasons explained above, although the first full day of Illinois’ stay-at-home order was day March 22, I used March 24 as the treatment date for all of the synthetic control analyses of cases per capita and March 31 as the treatment date for all of the synthetic control analyses of deaths per capita.

Results

Table 3 summarizes the results of all eight analyses that I conducted. I included the regressions in the table, although I conducted the regressions mainly to identify predictor variables for the nearest neighbor matching and synthetic control matching; the results from the state-level regression, in particular, should not be given much weight because of the small sample size. All eight methods estimated that Illinois’ stay-at-home order reduced the number of COVID-19 cases per capita in downstate Illinois through May 3. That effect was statistically significant in only three of those analyses, but the fact that the effect was so consistently negative strongly suggests that Illinois’ stay-at-home order did reduce the number of cases in downstate Illinois by 12 to 42 percent. By contrast, seven of the eight methods estimated that the stay-at-home order actually increased the number of deaths per capita in downstate Illinois, but none of those effects was statistically significant. Therefore, Illinois’ stay-at-home order may have been at least partly responsible for the fact that downstate Illinois had 46% fewer cases than the comparison states on May 3, but other factors are probably mainly responsible for the fact that downstate Illinois had 41% more deaths than those states on May 10.

Table 3. Summary of analyses of effects of Illinois’ stay-at-home order on COVID-19 cases and deaths in downstate Illinois and the five comparison states

Discussion

Why might Illinois’ stay-at-home order have reduced the number of cases in downstate Illinois, but not the number of deaths? One possible explanation would be if downstate Illinois had tested less aggressively than the comparison states. In that case, downstate Illinois may simply have detected fewer less severe cases and may have underreported its cases to a greater extent than the comparison states. Two indicators of how aggressively a state is testing are the number of tests conducted per capita and the percentage of positive tests. States that tested less aggressively might have a lower number of tests per capita and a higher percentage of positive tests. Table 4 shows testing data for downstate Illinois and the 5 comparison states through May 3. Downstate Illinois conducted 5 percent fewer tests per capita than the comparison states, but had a 37 percent lower positive test rate than those states. So, the comparison is mixed, but it does not appear that downstate Illinois had tested significantly less aggressively than the comparison states through May 3.

Table 4. COVID-19 testing data for downstate Illinois and the five comparison states

A second possible explanation would be if the comparison states had a greater percentage of people that were at risk of serious complications from COVID-19 because of age or preexisting medical conditions. The Kaiser Family Foundation (https://www.kff.org/global-health-policy/issue-brief/how-many-adults-are-at-risk-of-serious-illness-if-infected-with-coronavirus/) calculated the percentage of at-risk adults in each state — people older than age 64 and other adults with certain preexisting conditions such as heart disease, chronic obstructive pulmonary disease, uncontrolled asthma, diabetes, or a body mass index greater than 40. I used their methods to calculate that percentage for downstate Illinois. Overall, about 26% of downstate Illinois’ population (about 35% of the adult population) is at higher risk due to age or preexisting conditions, which is similar to the percentages for Iowa, Nebraska, North Dakota, and South Dakota and less than the percentage for Arkansas. So, downstate Illinois’ population as a whole does not seem more susceptible to the virus than the populations in the other states.

A third possible explanation would be if a greater percentage of at-risk people actually became infected in downstate Illinois than in the comparison states, even though the overall populations in the two groups have similar at-risk percentages. In particular, nursing facility residents have comprised a large percentage of COVID-19-related deaths in many states. So, if downstate Illinois had a greater percentage of nursing facility deaths, whether because of inferior procedures or simply due to bad luck, that could help to explain why Illinois’ stay-at-home order did not reduce the number of deaths per capita, even though it reduced the number of cases per capita. I could not obtain the historical data through May 10, but through May 28, about 71 percent of the COVID-19-related deaths in downstate Illinois were nursing facility-related (https://www.dph.illinois.gov/covid19/long-term-care-facility-outbreaks-covid-19). Among the five comparison states, those percentages were 42 percent for Arkansas, 54 percent for Iowa, and 55 percent for Nebraska; North Dakota and South Dakota did not reported the number of nursing home-related deaths at that time (https://freopp.org/the-covid-19-nursing-home-crisis-by-the-numbers-3a47433c3f70). So, it does appear that downstate Illinois’ percentage of nursing facility-related deaths was higher than average. If that percentage was 20 points higher than the average for the five comparison states, that difference could explain all of the 46 percent difference in COVID-19 deaths per capita summarized in Table 2. In other words, without considering nursing facility-related deaths, downstate Illinois’ COVID-19 death rate through May 10 may have been similar to the rate in the comparison states. But the fact that the nursing facility-related death rate may have been twice as high in downstate Illinois as it was in the control group states is still concerning.

What should we conclude from this entire analysis? Illinois’ stay-at-home order seems to have had the desired effect on mobility in downstate Illinois, as its residents reported a smaller increase in visits to parks and larger decreases in visits to retail and recreation places and workplaces than residents of the five comparison states. The stay-at-home order may also have had the desired effect on reducing the number of COVID-19 cases in downstate Illinois, as it experienced 41 percent fewer cases per capita than the comparison states through May 3 and I estimated that the stay-at-home order reduced the number of cases per capita by 12 to 42 percent, so at least part of that 41 percent difference was likely due to the stay-at-home order. However, unexpectedly, downstate Illinois experienced 46 percent more COVID-19 deaths per capita through May 10 than the comparison states and that difference may have been especially pronounced for nursing facility-related deaths. Considering that deaths are the most important and most reliable indicator of the spread of the pandemic, it would be difficult to conclude that Illinois’ stay-at-home order was effective in significantly reducing the health impact of the pandemic in downstate Illinois, as compared with the five states without stay-at-home orders.

That conclusion does not suggest, of course, that no social distancing measures were needed in downstate Illinois. All five comparison states that did not enact stay-at-home orders did enact other social distancing measures to varying degrees. Of those five states, Iowa had the strictest social distancing measures, as most nonessential businesses were closed, restaurant dining rooms were closed, and gatherings of more than 10 people were prohibited. Indeed, Dr. Anthony Fauci said that Iowa’s restrictions were “functionally equivalent” to a stay-at-home order. However, some of those five states had much less strict measures, as many non-essential businesses remained open in Arkansas, North Dakota, and South Dakota. South Dakota, in particular, probably had the least strict measures among the 48 contiguous states, as restaurant dining rooms also remained open and there were no restrictions on gathering sizes. Therefore, downstate Illinois could perhaps have followed a less restrictive course, with no stay-at-home order and with some nonessential businesses allowed to remain open, without having a significant increase in COVID-19 deaths.

Categories
COVID-19

COVID-19 Case Projections for Illinois Counties

(Updated through July 29)

(I have stopped updating this post, because my model is not as useful in predicting the course of the pandemic in Illinois counties, now that different states have taken very different approaches to managing the pandemic, with widely varying results).

Many researchers have created models with projections for COVID-19 cases, hospitalizations, intensive care unit stays, and/or deaths. Some of the most widely cited are models created by researchers at the University of Washington Institute for Health Metrics and Evaluation (https://covid19.healthdata.org/united-states-of-america), the University of Pennsylvania Perelman School of Medicine (https://penn-chime.phl.io/), the Los Alamos National Laboratory (https://covid-19.bsvgateway.org/), and a diverse team of epidemiologists and data scientists under a project called Covid Act Now (https://covidactnow.org/). Many of those models are excellent and have been utilized by federal, state, and local policymakers to help guide their responses to the pandemic. I do, however, have three concerns about many of those models. First, most models require the researchers to make some important assumptions about parameters such as the number of people to whom each infected person can transmit the virus, the number of days that a person is infectious, and the effects of various types of social distancing measures on transmission. Second, many models are basing their assumptions on data from other countries, such as China, Italy, and Spain, but the pandemic may play out differently in the United States than it has in other countries. And third, most models are designed to produce national estimates, which is challenging, because the pandemic is at different stages in different parts of the country. Due partly to these issues and partly to more basic methodological differences, current models are often producing very different projections from one another (https://projects.fivethirtyeight.com/covid-forecasts/).

I have created a model that relies on the fact that some parts of the country are further along the growth curve of the pandemic to produce COVID-19 case projections for Illinois counties. I use data only from the 48 contiguous states (and the District of Columbia), with the expectation that those states provide a better comparison for Illinois than foreign countries do. And I make no assumptions about the virus or the effects of social distancing on transmission of the virus; instead, the model’s projections are based only on the actual data from the 48 states.

My model is a multilevel nonlinear growth model using county-level data from the New York Times on cases reported each day in each county; I did not include data from New York City, both because New York City seems to be a unique case in terms of the pandemic’s spread and because data from New York City are reported only at the city level in the New York Times data, and not at the county level. The fixed portion of my model estimates the shape of the growth curve in total cases per 100,000 people using a fractional polynomial. The random portion of my model includes both random intercepts and random slopes at the state level and at the county level; the random slopes at both levels are also fractional polynomials. So, the overall shape of the growth curve is determined by a combination of national, state, and county trends; the model has allowed state and county trends to have more influence on the shape of the growth curve over time. The model used to also adjust for several demographic variables, but those variables have become less important over time as more days of case data have become available, so each county’s growth curve is now determined entirely by national, state, and county case data over time.

Of course, my model still has many of the usual limitations associated with these models. First, the number of reported cases is still believed to underestimate the actual number of COVID-19 infections in most counties, so relying on those data is inherently problematic. Second, there have been many reported clusters of COVID-19 infections in places such as nursing homes and prisons, which are virtually impossible to predict. Finally, there is typically a lot of uncertainty in making projections and my model is no exception to that rule. I do not include confidence intervals for my point estimates below because those intervals are so wide that they are of little value. With those limitations in mind, the table below includes the actual number of total cases in each Illinois county through July 29 and the projected number of total cases for each county through September 30.

The following chart shows the actual number of new daily cases reported in Illinois through July 29 and the number of new daily cases projected by my model after that date through September 30. My model projects that the number of new daily cases in Illinois will continue to increase through at least the end of September.

How do the results of my model compare to the results from other models? The University of Washington model does not provide publicly-available county-level estimates, but that model projects that the number of new daily cases in Illinois will decrease slightly through the end of September, before beginning to increase again (https://covid19.healthdata.org/united-states-of-america/illinois). The University of Pennsylvania model does not provide publicly available estimates for Illinois. The Los Alamos model also projects that the number of new daily cases in Illinois will decrease slightly through at least September 9 (https://covid-19.bsvgateway.org/). The Covid Act Now model provides estimates of projected hospitalizations for both the state of Illinois and individual counties in Illinois and projects that the number of hospitalizations in Illinois will decrease slightly through the end of August (https://covidactnow.org/us/il). So, all of those other models disagree with my model and project that, despite the increase in daily cases over the last couple of weeks, that new daily cases will instead decrease over the next month. I hope that those projections are correct, but, unfortunately, my model’s projection of a continuing increase in cases seems more realistic at this point.

Focusing on the downstate Illinois area, following are charts showing the actual and projected number of cases for several different health service areas. As for the state of Illinois as a whole, the growth curves for all of these health service areas are projected to continue to increase through at least the end of September.

I will continue to update this model occasionally. If you have any questions about the model, please feel free to contact me at grein3@uis.edu.