Categories
COVID-19

Effect of K-12 instruction types on reported COVID-19 cases and deaths in Illinois counties

(Updated on October 21)

Summary

Few decisions made by state and local governments in response to the coronavirus pandemic have affected families as much as decisions about K-12 instruction types – whether to provide in-person instruction, online-only instruction, or a hybrid of in-person and online instruction. Decisions about instruction types this fall have varied widely across states, counties, and school districts, partly because of differences in COVID-19 case metrics and partly for other reasons, including political differences. Despite this variation, it is difficult to determine exactly how much differences in instruction types have contributed to the spread of the pandemic, mainly because the instruction types were not randomly assigned to schools or districts, because many schools and districts have changed instruction types over time in response to changing community conditions, and because other factors that affect the spread of the pandemic may also have changed over time.

I estimated the effect of different K-12 instruction types on the spread of the pandemic in Illinois counties by comparing the average number of new daily reported COVID-19 cases and deaths over a three-week “post-treatment” period from August 24 to September 13, with the average number of new daily reported cases and deaths over a three-week “pre-treatment” period from August 3 to August 23. I compared three groups of counties based on the instruction type that a majority of county students used to start the school year. I used a synthetic control method that matched each county in each group with a combination of counties from each other group that were as similar as possible to it.

My results suggest that in-person instruction contributed significantly more to increases in the number of reported cases than either hybrid instruction or online-only instruction: having a majority of county students in hybrid districts may have resulted in about an 18% to 30% reduction in the number of new cases over the period from August 24 to September 13, as compared with having a majority of county students in in-person districts; and having a majority of county students in online-only districts may have resulted in about a 29% to 45% reduction in the number of new cases over that period, as compared with having a majority of county students in in-person districts. However, my results suggest that there was not a significant difference between hybrid and online-only instruction in contributing to increases in the number of reported cases. None of the differences in instruction types appear to have contributed significantly to increases in the number of reported deaths.

Introduction

In Illinois, as in almost all other states, many K-12 students did not return to their classrooms this fall, at least not on a full-time basis. An Education Week chart (https://www.edweek.org/ew/section/multimedia/map-covid-19-schools-open-closed.html) shows that only four states required schools to provide at least some in-person instruction, while two states required schools to provide only online instruction. In the remaining forty-four states, including Illinois, the type of instruction being provided varied by district or school. The Illinois State Board of Education (ISBE), in particular, left decisions about the type of instruction largely up to the 852 school districts in Illinois, although it did require districts to provide online instruction to any students whose parents requested it (https://www.isbe.net/Documents/Messsage-07232020.pdf). As a result, the type of instruction being provided has varied by district, with some districts providing only online instruction, some providing mainly in-person instruction, and some providing mainly a hybrid of online and in-person instruction. In some districts, the type of instruction being provided has varied by grade, with younger students receiving in-person or hybrid instruction and older students receiving online instruction.

The ISBE surveyed school districts in July about their plans for instruction to start the school year and posted the results of that survey in a dashboard on its website (https://www.isbe.net/coronavirus). According to that dashboard, 31% of districts (with 1,238,000 students) planned to provide only online instruction, 28% of districts (with 158,000 students) planned to provide mainly in-person instruction, and 41% of districts (with 527,000 students) planned to provide mainly hybrid instruction. Most Chicago-area districts and some of the larger downstate districts planned to provide only online instruction and most of the larger downstate districts planned to provide mainly hybrid instruction; districts that planned to provide mainly in-person instruction were mostly smaller downstate districts.

Like many social distancing restrictions that state and local governments imposed in response to the pandemic, decisions by school districts to keep schools closed have been controversial. Parents in many communities have protested to return to in-person instruction (https://chicago.cbslocal.com/2020/09/08/hundreds-turn-out-in-west-suburbs-to-protest-remote-learning-in-schools/), while teachers in some communities protested to continue online-only instruction (https://wgntv.com/news/coronavirus/ctu-holds-protest-calls-for-all-remote-learning/). A Pew survey in August found an important partisan divide regarding school reopening, as 36% of people who are or lean Republican said that schools should provide only in-person instruction and only 13% of those people said that schools should provide only online instruction, while 41% of people who are or lean Democratic said that schools should provide only online instruction and only 6% of those people said that schools should provide only in-person instruction (https://www.pewresearch.org/fact-tank/2020/08/05/republicans-democrats-differ-over-factors-k-12-schools-should-consider-in-deciding-whether-to-reopen/).

An important question, of course, was whether students could return to classrooms in a way that was safe for students, teachers, other school employees, and their families. In July, the Centers for Disease Control and Prevention (CDC) recommended that schools be reopened this fall, based on its conclusions that COVID-19 poses low risks to children and that children are not likely to be major contributors to the spread of the virus (https://www.cdc.gov/coronavirus/2019-ncov/community/schools-childcare/reopening-schools.html). However, the CDC’s recommendations were issued after President Trump and Education Secretary DeVos had stated that schools should be reopened and had threatened to withhold federal funding from districts that didn’t reopen, which led some people to question the basis of the CDC’s recommendations (https://www.washingtonpost.com/education/2020/07/28/democratic-lawmakers-probing-whether-cdc-guidelines-reopening-schools-were-influenced-by-political-pressure/).

Because almost all states closed all of their public schools at about the same time this spring during the start of the pandemic, it was almost impossible to determine what effect those school closings had on the spread of the pandemic. To determine the effects of a policy, there needs to be some variation in that policy, either across geographic units such as states, counties, or districts or across time within the same geographic units. Some researchers attempted to estimate the effect of the school closures this spring on the spread of the pandemic by comparing changes in the spread within states or counties before and after the closures occurred, but because so many other social distancing restrictions were being imposed at the same time, it was very difficult to disentangle the effect of the school closures from the effects of those other measures. However, because there was variation in the instruction types that Illinois school districts used to start the school year this fall, with some districts providing only online instruction, some providing mainly in-person instruction, and some providing mainly hybrid instruction, we can start to estimate the effects of these different instruction types on the spread of the pandemic.

Data Used

For data on school district instruction types, I started with the data on the ISBE dashboard, which stated that it had been updated through September 21. However, I observed that some information on the dashboard was not accurate. For example, Springfield District 186 is listed as providing hybrid instruction on the dashboard. But District 186 changed its plans in mid-August, after it had submitted its survey to ISBE, and has been providing only online instruction since the beginning of the school year. Therefore, I updated the ISBE dashboard data for the 200 largest districts in Illinois and, to the extent they weren’t included among the 200 largest districts, for the largest district in each county, by searching district websites and news sources. Many districts seem to have changed their plans in August, after they had submitted their surveys to ISBE. Some other districts may have misdescribed their instruction type to ISBE, as they characterized instruction that is in-person for all students except students whose parents requested online instruction as hybrid, when it should be classified as in-person. In some cases, school districts have changed instruction types since the beginning of the school year, with some districts switching to online-only instruction in response to school or community outbreaks and other districts starting to open schools for in-person instruction as community metrics improved. I coded the districts based on the instruction type that they were using at the beginning of the school year. When a district had different instruction types for different grades, I coded the district with the least restrictive instruction type being used (in-person is the least restrictive type, then hybrid, then online-only).

Because COVID-19 case and death data are available only at the county level, not at the school district level, all of my analyses are at the county level. I first divided the counties into three groups: counties where a majority of the students attended districts with primarily in-person instruction, counties where a majority of the students attended districts with primarily hybrid instruction, and counties where a majority of the students attended districts with only online instruction. I excluded Cook County, because of its unique characteristics, and I excluded six counties with relatively large numbers of college students (Champaign, Coles, DeKalb, Jackson, McDonough, and McLean Counties), because the resumption of college classes contributed to case spikes in many college towns. Of the 95 remaining counties, 41 were majority in-person counties, 32 were majority hybrid counties, and 17 were majority online-only counties; 5 counties did not have a majority of students in any of the three categories. The table below shows the counties in each of these three groups and the percentage of students in each type of district for each county.

Analysis Methods and Results

To determine whether there was a relationship between these three groups of counties and the number of newly reported COVID-19 cases or deaths, I used a method called synthetic control matching to conduct a series of analyses, matching counties from each group with counties from each other group. I matched the counties on ten demographic variables that were important predictors either of the number of newly reported cases or deaths or of which group the county was in: median household income, poverty rate, population density, size (land area of the county), median age, the percentage of residents who are Hispanic, the percentage of residents who are black, the percentage of residents who are Native American, the percentage of residents who have attended at least some college, and the county international migration rate (which measures the net percentage of people moving into or out of the county to or from other countries); I also included a pretreatment average of the number of daily reported cases or deaths as a predictor variable. With two groups of units (treated units and control units), synthetic control matching constructs a synthetic control unit for each treated unit by finding a weighted combination of the control units that matches the treated unit as closely as possible on the pretreatment averages of the predictor variables. The advantage to synthetic control matching is that a weighted combination of control units can often provide a better match for a treated unit than any individual control unit or even than an average of two or more control units.

Many Illinois schools started a week or two later than usual this year, with most schools starting sometime between August 24 and September 2. Therefore, I used August 24 as the treatment date in my analyses. Although a few schools started earlier than August 24, it usually takes at least a few days for COVID-19 cases to be detected and reported. Therefore, I would not expect school reopenings to noticeably affect reported COVID-19 case numbers until at least August 24; I would not expect school reopenings to noticeably affect reported COVID-19 death numbers until at least a few more days after August 24.

I conducted three synthetic control analyses each for reported cases and reported deaths: an analysis that compared majority in-person counties with majority hybrid counties, an analysis that compared majority hybrid counties with majority online-only counties, and an analysis that compared majority in-person counties with majority online-only counties. The table below shows the results of those six synthetic control analyses. The numbers in the table show the estimated effect of being in the more restrictive group on the number of reported cases or deaths per 100,000 people; negative numbers indicate that the more restrictive instruction type resulted in fewer cases or deaths. The asterisks in the table show whether the effect was statistically significant: three asterisks indicate that the effect was significant at a p-value of .01, meaning that it was at least 99% likely that there was an effect; two asterisks indicate that the effect was significant at a p-value of .05, meaning that it was at least 95% likely that there was an effect, and one asterisk indicates that the effect was significant at a p-value of .10, meaning that it was at least 90% likely that there was an effect; estimated effects without any asterisks were not statistically significant.

*** p < .01; ** p < .05; * p < .10

The second column of the table shows that majority hybrid counties had significantly fewer cases per 100,000 people than their synthetic control units of majority in-person counties from August 29 to September 10. Similarly, the sixth column of the table shows that majority online-only counties had significantly fewer cases per 100,000 people than their synthetic control units of majority in-person counties from August 24 to September 14 (except for a few days near the end of that period, when the effects were not quite statistically significant). However, there was no significant effect on cases per 100,000 people for majority online-only counties as compared with majority hybrid counties (except for one day when the effect was barely statistically significant). And there were no significant effects on deaths per 100,000 people in any of the comparisons.

For the comparison of cases in majority in-person counties and majority hybrid counties, the effect followed a U-shaped pattern, with no effect at the beginning of the period, a statistically significant negative effect in the middle of the period, and no effect at the end of the period. This U-shaped effect was not unexpected. It is not surprising that the difference between in-person and hybrid instruction did not begin to affect the number of reported cases for several days. It is also not surprising that, after several days, majority hybrid counties had significantly fewer reported cases than their synthetic control units of majority in-person counties, because in-person instruction generally involves having almost all students at school at the same time, while hybrid instruction usually means having less than half of students at school at any particular time. Finally, it is not surprising that the effect disappeared toward the end of the period, because many in-person districts that experienced school or community outbreaks changed quickly to hybrid or online-only instruction and some hybrid districts with favorable community metrics changed to in-person instruction. So, there was a significant amount of crossover between treated and control counties by the end of the period. The comparison of cases in majority in-person counties and majority online-only counties also showed a U-shaped effect pattern, although even on the first day of the period, majority online-only counties already had significantly fewer reported cases than their synthetic control units of majority in-person counties.

It is also not surprising that there was not a significant effect on deaths in any of the comparisons. It is generally more difficult to find significant effects on COVID-19 deaths than it is on cases, because the timing from infection to death varies much more than the timing from infection to confirmation of a positive case. And in this case, the population of students, teachers, and other employees at K-12 schools are generally not in the high-risk groups for severe complications from the virus as some populations are, such as nursing facility residents. Although there are undoubtedly some students, teachers, and other school employees who are in high-risk groups or who have immediate family members in high-risk groups, many of those people likely chose not to participate in in-person classes in in-person or hybrid districts.

The most unexpected result was that there was not a significant effect on reported cases in the comparison of majority hybrid counties and majority online-only counties. That result suggests that hybrid instruction did not contribute significantly to the spread of the pandemic. As noted above, hybrid instruction generally involves having fewer than half of the students in each class attend school at any particular time, which may have allowed the students and teachers to maintain an adequate distance to prevent significant transmission of the virus. Of course, screening and contact tracing likely also contributed, by keeping potentially infected people out of the classrooms and quickly controlling any outbreaks that did occur through quarantines.

I also conducted analyses comparing the three groups of counties using other analysis methods, such as difference-in-differences, difference-in-differences with kernel propensity score weights (PSW), propensity score matching, and propensity score regression. To have common time periods for all of the analyses, I compared the results from a three-week posttreatment period from August 24 to September 13 with the results from a three-week pretreatment period from August 3 to August 23. The effect estimates from the other methods were generally smaller than the synthetic control estimates and were almost never statistically significant. However, the synthetic control results were more consistent to different model specifications than the results from the other methods, so I have greater confidence in the synthetic control results. The table below summarizes the results for reported cases from the preferred version of each of these methods; the propensity score-based methods were not able to produce an estimate for the comparison of majority in-person counties and majority online-only counties.  The results for reported deaths were again small and never statistically significant, so I did not include them in the table.

** p < .05; * p < .10

So, having a majority of county students in hybrid districts may have resulted in about 4 to 8 fewer new daily cases per 100,000 people over that three-week period from August 24 to September 13 (an 18% to 30% reduction in the number of new cases), as compared with having a majority of county students in in-person districts; and having a majority of county students in online-only districts may have resulted in about 6 to 12 fewer new daily cases per 100,000 people over that period (a 29% to 45% reduction in the number of new cases), as compared with having a majority of county students in in-person districts. There again was not a significant difference between majority hybrid districts and majority online-only districts, although the fact that all of the estimates were negative suggests that online-only instruction may have had some small additional advantage over hybrid instruction in terms of reducing the spread of the pandemic, with perhaps 1 to 4 fewer new daily cases per 100,000 people over that period for majority online-only districts.

Limitations

There are several factors that could affect the accuracy of my estimates. The reliability of reported COVID-19 case data has improved as testing capacity has improved, but a large percentage of infections are likely still going undetected and unreported. The ISBE data on instruction types were not completely accurate and, even though I updated them for the 200 largest districts in Illinois and some additional districts that are the largest districts in their counties, there were likely still inaccuracies in my data. Even if the data were accurate, there is a significant amount of spillover between the three groups of counties, as people regularly travel from county to county and some students and teachers may even attend or work at a school in a different county than the county in which they live. And, as discussed above, there has also been a lot of crossover among districts and probably also among these three groups of counties, as districts have changed instruction types in response to changing school or community conditions.

Although I controlled for many important demographic differences between counties, I did not control for differences at the district or school level, which could also have affected my results. In particular, there may be important differences in facilities or resources between districts that started the school year with hybrid instruction and districts that started with online-only instruction, such that the online-only districts would not have been able to offer hybrid instruction as safely as my results suggest that the hybrid districts did. Also, my sample sizes were relatively small, especially for the number of majority online-only counties, which may have prevented me from finding statistically significant effects, especially for the comparison of majority hybrid counties and majority online-only counties. Finally, there may be other important factors that changed from August to September that affected my results, such as if some counties tightened or relaxed other social distancing measures. In particular, if there were differences in how fall sports and other extracurricular activities (either school-based or nonschool) were conducted, that could have affected my results; it is quite possible that counties or districts that were less restrictive in school instruction types were also less restrictive in other activities.

Conclusion

Overall, though, it appears that in-person instruction contributed significantly more to increases in the number of reported cases than either hybrid instruction or online-only instruction and that there was not a significant difference between hybrid and online-only instruction in contributing to those increases. None of the differences in instruction types appear to have contributed significantly to increases in the number of reported deaths. It will continue to be very difficult to accurately estimate the effect of different K-12 instruction types on the spread of the pandemic and there are significant limitations to my research, as I discussed above. However, it is important to start attempting to estimate these effects, so that school districts can make informed decisions about their instruction types. Of course, those decisions also depend on many other factors. Districts need to consider the academic and social effects of the different instruction types on children; the risks associated with the virus for children, teachers, other school employees, and their families; the effects of different instruction types on parents’ work schedules; and the financial costs for schools to safely reopen buildings to students. Research on any of these issues can help to inform those decisions.

It is important, however, that districts do not base their decisions solely on the current situations in their communities; they should also consider how their decisions will affect those situations. The fact that the community may have a higher number of cases or a higher positivity rate, in general, than a predetermined target does not necessarily mean that school must be conducted only online. The results from Illinois counties that I discussed above suggest that hybrid instruction may not have significantly contributed to additional increases in the number of new cases in the community, as compared with online-only instruction. A recent study from Spain and anecdotal evidence from Utah and other states have also indicated that school reopenings may not have contributed significantly to the spread of the pandemic (https://www.nprillinois.org/post/were-risks-reopening-schools-exaggerated). My results do not go that far, as they do suggest that full in-person instruction may have contributed to increases in reported cases in Illinois counties.

If you have any questions about this research, please contact me at grein3@uis.edu.