Foreword
To date, most of what we’ve heard in the news about pandemic learning loss has been specific to public schools. ERB’s new analysis of independent school data is a welcome addition to our shared understanding. Five years past the onset of COVID-19, this report helps to quantify some of the impact of the pandemic on our sector as a whole and to situate us within our recovery. It also proposes how we might build on lessons learned.
Ultimately, this report’s conclusion is hopeful: that within this time frame, independent school students at all levels of achievement have largely “caught up” in their pace of knowledge mastery across content areas and restored their learning patterns. Of course, behind this data are the tens of thousands of teachers who put in the creativity, care, and countless hours needed to help students achieve this level of learning recovery – and the many leaders whose support, decision-making, and focus on mission paved the way. In other words, the independent school model held up in the face of our biggest collective challenge, a reminder that resilience is a critical characteristic not only for students, but also for institutions.
At NAIS, we are grateful for our strong partnership with ERB. This publication is a testament to the power of working together to strengthen the many intersecting elements of independent schools that allow them to thrive.
Debra P. Wilson
President, NAIS
Executive Summary
The impact of the COVID pandemic on student learning was profound in all schools— public and private, secular and faith-based. The impact was especially great on those students whose academic performance was relatively weak before the pandemic since those are the students who most rely on teachers and other educational supports. Among NAIS member schools that are also ERB members, however, student learning has made a substantial recovery from the COVID disruption. Assessment data from the 2023-2024 school year show that the pace of student learning approximates pre-pandemic rates of growth in ELA and has exceeded pre-pandemic growth in Math. These data demonstrate that concerns over long-term COVID impacts on student learning do not apply to students in independent schools.
Introduction
COVID-related disruption of normal social interaction in March 2020 will long remain seared in our collective memory. Workplaces were abandoned, hospitals and senior centers could not accept visitors, and schools closed their doors—all on short notice and with little idea how long those closures might last. While many areas of endeavor have since returned to normal, concerns linger over the long-term impact of the COVID disruption on students in their formative years. A National Assessment of Educational Progress (NAEP) report published in 2022 showed that:
“…Nine-year-old students, typically in fifth grade, experienced the largest-ever score decline in reading and the first-ever recorded score decline in math. Peggy Carr, Commissioner of the National Center for Education Statistics (NCES), was cited in USA Today as saying ‘It’s clear that COVID-19 shocked American education and stunted the growth of this age group of children.’”1
Commentary on COVID’s impact on education has been dominated by test results from students in public schools. In a series of studies dating back to late 2020, however, ERB has mined its database of Comprehensive Testing Program (CTP) results to understand the extent to which COVID-related trends affected student learning in independent and faith-based private schools.
The purpose of this paper is to summarize our findings on the COVID disruption among ERB members in the first 18 months of the pandemic. We will then contrast that disruption with the striking restoration of student learning rates among NAIS member schools beginning in the 2021-2022 school year and continuing to the present. The story told by our student learning data is one of effective adaptation and continued educational excellence displayed by administrators, teachers, and students alike.
The Initial Impact of COVID-19 on Student Learning
To assess the impact of Spring 2020 school closures on student learning, we analyzed results from the Comprehensive Testing Program (CTP) from a panel of students who took the test battery in Fall 2018, Fall 2019, and Fall 2020. Using data from all ERB member schools, we compared the test score growth rate from Fall 2018 to Fall 2019 (pre-COVID) with the growth rate from Fall 2019 to Fall 2020 (spanning the school closures in Spring 2020).2
Figure 1 demonstrates the impact of the transition to remote education on student learning. Data from Grades 2 through 7 are combined in this graph, though inspection of the results for each grade shows a great deal of consistency.
Figure 1
Year-Over-Year Learning Gain Among Independent School Students
COVID-related disruptions can be clearly seen to have reduced the rate of student learning during the Spring and Fall of 2020. Student growth from 2019 to 2020 in Mathematics, Reading Comprehension, and Writing Concepts & Skills was approximately 65 to 70% of the amount of growth exhibited by those same students the previous year. While significant, it is worth noting that a 30% loss of growth momentum was far less dire than results reported at the time based on national studies of statewide testing in public schools.
Figure 1 also shows that the impact on learning growth was greater in Quantitative Reasoning and Verbal Reasoning than in the CTP subject tests. Students had roughly half the amount of growth in reasoning skills from Fall 2019 to Fall 2020, compared to the growth shown by those same students one year earlier.
Reasoning skills are built over time by acquiring a depth of understanding that enables students to connect new material to what they already know and to work through problems they have not yet encountered. That kind of deep engagement with the material—mastery rather than surface understanding of the facts—was disproportionately impacted when students abruptly lost daily contact with teachers and peers in the classroom.
These findings raise the question of whether the COVID disruption was greater on some groups of students than others. To address this, we examined the rate of learning growth among the strongest and weakest students by dividing our population into the bottom 25% in test performance (“Low”), the middle 50% (“Middle”), and the top 25% (“High”) based on their test scores one year earlier, in 2017-2018.
Figure 2
Test Score Growth by Level of Student Performance
Between Fall 2018 and Fall 2019, students in the Low group showed the greatest degree of test score gain (14.6%), followed by the Middle group (9.0%) and the High group (6.5%). This is the usual pattern of growth from one year to the next. Some students in the low-performing group one year are ready to accelerate their growth, aided by the fact that teachers and learning specialists are focused on their needs. Students in the Low group also have more opportunities for growth since there is more material still to be learned at their grade level.3
In the pandemic-disrupted Spring and Fall of 2020, however, this typical pattern of growth changed dramatically. The amount of test score growth over that period was almost indistinguishable among the three groups, ranging from 6.3% in the lowest group to 5.9% in the highest group. While consistency in learning growth across student performance levels may at first seem gratifying, it is important to note that the lowest-performing student group was actually the most disadvantaged compared to the growth they would typically have experienced from one year to the next. The virtual and hybrid learning models that schools were forced to adopt on short notice thus had the greatest negative impact on those students who were already in the lowest achievement group.
This pattern is exaggerated in the verbal and quantitative reasoning tests, with top-quartile students demonstrating slightly more learning growth than those in the bottom quartile. The strongest students seem not to rely as much on teacher interaction to incorporate new knowledge into their reasoning process, thus enhancing their problem-solving success.
These findings outline the contours of the challenge faced by teachers and administrators as the COVID pandemic subsided and school attendance returned to normal. The central role of teacher-student interaction in the classroom needed to be re-established, with a particular emphasis on supporting the lowest achieving students and on supporting the development of reasoning skills.
Restoration of Student Learning Patterns Among NAIS Schools Since the COVID Disruption
We have now had three full academic years relatively free of COVID disruption. To see how students have responded, we compare student growth in NAIS member schools during the final pre-COVID year of 2018-2019 with post-COVID growth rates from 2021 through 2024.4
Figure 3 shows the results of CTP tests grouped in two different ways, contrasting Math tests with English Language Arts (ELA) tests and then regrouping those same tests to contrast subject tests with assessments of reasoning.
Figure 3
Comparing Test Score Growth Pre- and Post-COVID
Using the 2018-2019 learning growth results as our baseline for comparison, we can see that student growth in 2021-2022 exceeded pre-COVID growth in ELA and matched pre-COVID growth in Math. Growth in subject test areas exceeded pre-pandemic rates, a catch-up effect responsive to the slower growth between 2019 and 2021. Student growth in reasoning tests took longer to be fully reestablished, exceeding pre-pandemic rates in the 2022-2023 school year. Stepping back to look at the big picture conveyed by Figure 3, one would never know that schools experienced a massive disruption in the Spring of 2020. This is a remarkable outcome that speaks volumes about the resilience of the learning environment in independent schools.
How successful have independent schools been in supporting the lower-achieving students who were most impacted by school closures during the first phase of the pandemic?
Figure 4 shows a re-establishment of the stair-step pattern in which lower achieving students experience higher rates of learning growth. Lower-achieving students may have been disproportionately disadvantaged during the height of the COVID disruption, an effect that lingered during 2021-2022, but by the 2022-2023 school year, they had re-established their pre-COVID pace of learning.
Figure 4 also shows that students in every achievement group—low, middle, and high —have been able to accelerate their pace of Mathematics learning compared to the pre-COVID standard. Among the strongest students, the acceleration of Mathematics learning growth has been especially striking.
Figure 4
Test Score Growth by Level of Performance
Figure 5 contrasts the growth of subject knowledge and reasoning skills among students in the bottom quartile of performance, those in the middle two quartiles, and those in the top quartile. Students in every quartile of performance have fully recovered their pre-COVID rates of growth in both subject and reasoning tests, bolstered particularly by the strong growth rates in Math. Compared to the pre-COVID benchmark, students in the top quartile have markedly accelerated their growth rates both in subject knowledge and in reasoning ability.
Figure 5
Test Score Growth by Level of Performance
Two factors may be contributing to the higher rates of learning growth by students in the top quartile of achievement. One is that these students have a catch-up opportunity created by the initial COVID impact. High-achieving students are usually near the ceiling in growth potential because of their existing mastery of grade-level material. With the impact on learning experienced during 2020-2021, however, there was an increased opportunity to accelerate subsequent learning growth.
It is also possible that the most capable students flourished precisely because of the habits of greater independence required of them during the initial phase of the pandemic. The 2022 NAEP study of academic performance among 9-year-old students nationwide was paired with a survey in which participants were asked several questions about their educational experience during COVID. The NAEP report concludes that:
“Higher-performing students reported more confidence in their ability to recognize when they don’t understand something they are learning, ask for help when they need it, and find learning resources online to learn more about something they don’t understand, compared to their lower-performing peers.”
The COVID-19 pandemic taught us many things about work, community, and human nature. One of those lessons specifically applicable to education may be the extent to which the most capable students flourish when given greater independence and responsibility for their learning. It is to the great credit of independent schools that their strongest students have continued an accelerated rate of learning in the years since the pandemic ended.
Conclusion and Action Implications
These findings on student learning growth suggest a number of take-away lessons. First and most importantly, independent schools have been highly successful in reestablishing pre-COVID rates of learning growth. The initial COVID impact on learning growth was greater among weaker students and greater in the areas of verbal and quantitative reasoning than in the acquisition of subject knowledge. Assessment results from the last three years show that these impacts of COVID disruption on learning growth have been substantially erased.
The accelerated learning growth experience of the highest-performing students during the last three years presents NAIS member schools with an opportunity to reflect on ways they might capitalize on the habits of independent learning these students have developed. The goal here would be to identify the strategies that have enabled the strongest students to achieve unprecedented rates of learning growth and to adapt those strategies to benefit all students.
COVID-19 created a disruption in school routines that was without precedent in living memory. ERB’s assessment data make clear that NAIS member schools can take pride in their effectiveness in sustaining an environment of educational excellence despite the challenges of the pandemic.
1 USA Today, September 1, 2022. For the NAEP results, see https://www.nationsreportcard.gov/highlights/ltt/2022
2 Nearly 2 million subject-specific CTP tests are typically taken each year by over 250,000 students drawn from over 1,000 schools, making any analysis of learning growth trends highly reliable. During fall 2020, however, fewer students took CTP tests than usual. Those who did so often took the tests at home, delivered online with their teachers serving as remote proctors. To ensure comparability, we examined score trends from 2018 and 2019 for this much smaller sample and found that they are representative of the larger universe of CTP test takers.
3 This is an example of what statisticians call “regression to the mean”—a tendency of those who are low in one measurement to be higher in a subsequent measurement, while those who are highest in the first measurement will tend to be less high in the next. We find this consistently across each of the CTP subtests.
4 Over 1400 of the 1500+ ERB member schools are also members of NAIS, and a majority of those schools use CTP as one of their assessments of student learning. Annual growth results in Figures 3-5 are based on test results from between 176,000 and 259,000 students, depending on the year.
ABOUT THE AUTHOR
Dr. Tom Rochon
Dr. Thomas R. Rochon became president of ERB in 2017. His previous career in higher education included faculty appointments at Princeton University and Claremont Graduate University, a postdoctoral fellowship at Stanford University, and a year as a Fulbright Scholar in Japan. Administratively, Tom has served as dean and provost at Claremont Graduate University in California, provost at the University of St. Thomas in Minnesota, and president of Ithaca College in New York.
Within the world of educational assessment, Tom previously served as executive director of the GRE testing program at the Educational Testing Service (ETS). Tom’s scholarly books and articles include Mobilizing for Peace: The Antinuclear Movements in Western Europe (Princeton University Press, 1988) and Culture Moves: Ideas, Activism and Changing Values (Princeton University Press, 1998). He received his bachelor’s degree, master’s degree, and Ph.D. all in political science from the University of Michigan.
ABOUT THE AUTHOR
Aaron Shuman
Aaron Shuman is a technology professional with 25 years of experience in data-driven design and implementation. He has an educational background in statistics, psychology, and computer science with degrees from Yale University. In his current role at ERB, Shuman is the Director of Analytics and Insights, with a focus on providing ERB members with a better understanding of their students’ performance over time. His specific responsibilities include data integration and analysis for 360 Access, ERB’s data reporting platform.
