Executive Summary
The impact of the COVID pandemic on student learning was profound in all schools— public and private, secular and faith-based. The impact was especially great on those students whose academic performance was relatively weak before the pandemic since those are the students who most rely on teachers and other educational supports. Among CAIS member schools, however, student learning has made a complete recovery from the COVID disruption. Assessment data from the 2023-2024 school year shows that the pace of student learning exceeds pre-pandemic rates of growth, both in subject knowledge and in reasoning ability. These data demonstrate that concerns over long-term COVID impacts on student learning do not apply to students in CAIS member schools.
Introduction
COVID-related disruption of normal social interaction in March 2020 will long remain seared in our collective memory. Workplaces were abandoned, hospitals and senior centers could not accept visitors, and schools closed their doors—all on short notice and with little idea how long those closures might last. While many areas of endeavor have since returned to normal, concerns linger over the long-term impact of the COVID disruption on students in their formative years. A National Assessment of Educational Progress (NAEP) report published in 2022 showed that:
“…Nine-year-old students, typically in fifth grade, experienced the largest-ever score decline in reading and the first-ever recorded score decline in math. Peggy Carr, Commissioner of the National Center for Education Statistics (NCES), was cited in USA Today as saying ‘It’s clear that COVID-19 shocked American education and stunted the growth of this age group of children.’”1
Commentary on COVID’s impact on education has been dominated by test results from students in public schools. In a series of studies dating back to late 2020, however, ERB has mined its database of Comprehensive Testing Program (CTP) results to understand the extent to which COVID-related trends affected student learning in independent and faith-based private schools.
The purpose of this paper is to summarize our findings on the COVID disruption among ERB members in the first 18 months of the pandemic. We will then contrast that disruption with the striking restoration of student learning rates among independent schools in California beginning in the 2021-2022 school year and continuing to the present. The story told by our student learning data is one of effective adaptation and continued educational excellence displayed by administrators, teachers, and students alike.
The Initial Impact of COVID-19 on Student Learning
To assess the impact of Spring 2020 school closures on student learning, we analyzed results from the Comprehensive Testing Program (CTP) from a panel of students who took the test battery in Fall 2018, Fall 2019, and Fall 2020. Using data from all ERB member schools, we compared the test score growth rate from Fall 2018 to Fall 2019 (pre-COVID) with the growth rate from Fall 2019 to Fall 2020 (spanning the school closures in Spring 2020).2
Figure 1 demonstrates the impact of the transition to remote education on student learning. Data from Grades 2 through 7 are combined in this graph, though inspection of the results for each grade shows a great deal of consistency.
Figure 1
Year-Over-Year Learning Gain Among Independent School Students
COVID-related disruptions can be clearly seen to have reduced the rate of student learning during the Spring and Fall of 2020. Student growth from 2019 to 2020 in Mathematics, Reading Comprehension, and Writing Concepts & Skills was approximately 65 to 70% of the amount of growth exhibited by those same students the previous year. While significant, it is worth noting that a 30% loss of growth momentum was far less dire than results reported at the time based on national studies of statewide testing in public schools.
Figure 1 also shows that the impact on learning growth was greater in Quantitative Reasoning and Verbal Reasoning than in the CTP subject tests. Students had roughly half the amount of growth in reasoning skills from Fall 2019 to Fall 2020, compared to the growth shown by those same students one year earlier.
Reasoning skills are built over time by acquiring a depth of understanding that enables students to connect new material to what they already know and to work through problems they have not yet encountered. That kind of deep engagement with the material—mastery rather than surface understanding of the facts—was disproportionately impacted when students abruptly lost daily contact with teachers and peers in the classroom.
These findings raise the question of whether the COVID disruption was greater on some groups of students than others. To address this, we examined the rate of learning growth among the strongest and weakest students by dividing our population into the bottom 25% in test performance (“Low”), the middle 50% (“Middle”), and the top 25% (“High”) based on their test scores one year earlier, in 2017-2018.
Figure 2
Test Score Growth by Level of Student Performance
Between Fall 2018 and Fall 2019, students in the Low group showed the greatest degree of test score gain (14.6%), followed by the Middle group (9.0%) and the High group (6.5%). This is the usual pattern of growth from one year to the next. Some students in the low-performing group one year are ready to accelerate their growth, aided by the fact that teachers and learning specialists are focused on their needs. Students in the Low group also have more opportunities for growth since there is more material still to be learned at their grade level.3
In the pandemic-disrupted Spring and Fall of 2020, however, this typical pattern of growth changed dramatically. The amount of test score growth over that period was almost indistinguishable among the three groups, ranging from 6.3% in the lowest group to 5.9% in the highest group. While consistency in learning growth across student performance levels may at first seem gratifying, it is important to note that the lowest-performing student group was actually the most disadvantaged compared to the growth they would typically have experienced from one year to the next. The virtual and hybrid learning models that schools were forced to adopt on short notice thus had the greatest negative impact on those students who were already in the lowest achievement group.
This pattern is exaggerated in the verbal and quantitative reasoning tests, with top-quartile students demonstrating slightly more learning growth than those in the bottom quartile. The strongest students seem not to rely as much on teacher interaction to incorporate new knowledge into their reasoning process, thus enhancing their problem-solving success.
These findings outline the contours of the challenge faced by teachers and administrators as the COVID pandemic subsided and school attendance returned to normal. The central role of teacher-student interaction in the classroom needed to be re-established, with a particular emphasis on supporting the lowest achieving students and on supporting the development of reasoning skills.
Restoration of Student Learning Patterns Among CAIS Schools Since the COVID Disruption
We have now had three full academic years relatively free of COVID disruption. To see how students have responded, we compare student growth among CAIS schools that are ERB members during the final pre-COVID year of 2018-2019 with post-COVID growth rates from 2021 through 2024.4
Figure 3 shows the results of CTP tests grouped in two different ways, contrasting Math tests with English Language Arts tests and then regrouping those same tests to contrast subject tests with assessments of reasoning.
Figure 3
Comparing Test Score Growth Pre- and Post-COVID
Using the 2018-2019 learning growth results as our baseline for comparison, we can see that student growth was modestly higher in Math and markedly higher in ELA. One might be tempted to think of the 2021-2022 ELA results as being due to a “catch-up effect” in which students rebounded from the lower growth rates characteristic of the 18 months of COVID disruption. Results from the most recent school year, though, demonstrate that the acceleration of ELA learning was not temporary. This is a remarkable outcome that speaks volumes about the resilience of the learning environment in CAIS independent schools.
Remembering that reasoning skills were especially hard hit during the COVID disruption, we also grouped CTP tests into those focused primarily on subject knowledge–Math, Reading Comprehension and Writing Concepts & Skills—and those focused on ability to use subject knowledge to solve multi-step problems–Quantitative Reasoning and Verbal Reasoning.
A comparison of 2018-2019 learning growth with the school years beginning in 2021- 2022 shows a more rapid pace of student learning growth, in both subject and reasoning tests, in two of the three post-COVID years. Looking at the right side of Figure 3, one would never know that schools experienced a massive disruption in the Spring of 2020.
How successful have CAIS member schools
been in supporting the lower-achieving students
who were most impacted by school closures
during the first phase of the pandemic?
Figure 4 shows a re-establishment of the stair-step pattern in which lower-achieving students experience higher rates of learning growth. Lower-achieving students may have been disproportionately disadvantaged during the height of the COVID disruption, but by 2022-2023 they had reestablished their pre-COVID pace of learning, particularly in Mathematics.
Figure 4 also shows that students in the middle and high achievement groups have been able to accelerate their pace of learning compared to the pre-COVID standard, particularly in Mathematics. Acceleration of learning growth has been especially striking among the strongest students.
Figure 4
Test Score Growth by Level of Performance
Figure 5 contrasts the growth of subject knowledge and reasoning skills among students in the bottom quartile of performance, those in the middle two quartiles, and those in the top quartile. Students in the first three quartiles of performance have fully recovered their pre- COVID rates of growth in both subject and reasoning tests. Students in the top quartile have accelerated their growth rates both in subject knowledge and in reasoning ability. Students in the lowest quartile of performance, by contrast, have yet to match their pre-COVID rates of learning growth in reasoning skills. This is the singular area on which educators should give continued focus in order to remedy COVID-era deficits.
Figure 5
Test Score Growth by Level of Performance
We should not overlook the striking performance of top quartile students in recent years. Two factors may be contributing to the higher rates of learning growth by those students. One is that they have a catch-up opportunity created by the initial COVID impact. High achieving students are usually near the ceiling in growth potential because of their existing mastery of grade-level material. With the impact on learning experienced during 2020-2021, however, there was an increased opportunity to accelerate subsequent learning growth.
It is also possible that the most capable students flourished precisely because of the habits of greater independence required of them during the initial phase of the pandemic. The 2022 NAEP study of academic performance among 9-year-old students nationwide was paired with a survey in which participants were asked a number of questions about their educational experience during COVID. The NAEP report concludes that:
“Higher-performing students reported more confidence in their ability to recognize when they don’t understand something they are learning, ask for help when they need it, and find learning resources online to learn more about something they don’t understand, compared to their lower-performing peers.”
The COVID-19 pandemic taught us many things about work, community, and human nature. One of those lessons specifically applicable to education may be the extent to which the most capable students flourish when given greater independence and responsibility for their learning. It is to the great credit of California independent schools that their strongest students have continued an accelerated rate of learning in the years since the pandemic ended.
Conclusion and Action Implications
These findings on student learning growth suggest a number of take-away lessons. First and most importantly, California independent schools have been highly successful in reestablishing pre-COVID rates of learning growth. The initial COVID impact on learning growth was greater among weaker students and greater in the areas of verbal and quantitative reasoning than in the acquisition of subject knowledge. Assessment results from the last two years show that these impacts of COVID disruption on learning growth have been largely erased.
California independent schools should continue to focus on learning growth among lower-performing students, with an emphasis particularly on reasoning—the application of knowledge to solving problems.
At the same time, accelerated learning growth among the highest-performing students during the last three years presents California independent schools with an opportunity to reflect on ways they might capitalize on the habits of independent learning these students have developed. The goal here would be to identify the strategies that have enabled the strongest students to achieve unprecedented rates of learning growth and to adapt those strategies to benefit all students.
COVID-19 created a disruption in school routines that was without precedent in living memory. ERB’s assessment data make clear that California independent schools can take pride in their effectiveness in sustaining an environment of educational excellence despite the challenges of the pandemic.
1 USA Today, September 1, 2022. For the NAEP results, see https://www.nationsreportcard.gov/highlights/ltt/2022
2 Nearly 2 million subject-specific CTP tests are typically taken each year by over 250,000 students drawn from over 1,000 schools, making any analysis of learning growth trends highly reliable. During fall 2020, however, fewer students took CTP tests than usual. Those who did so often took the tests at home, delivered online with their teachers serving as remote proctors. To ensure comparability, we examined score trends from 2018 and 2019 for this much smaller sample and found that they are representative of the larger universe of CTP test takers.
3 This is an example of what statisticians call “regression to the mean”—a tendency of those who are low in one measurement to be higher in a subsequent measurement, while those who are highest in the first measurement will tend to be less high in the next. We find this consistently across each of the CTP subtests.
4 Over 80% of CAIS members also belong to ERB. Approximately 14,000 CAIS member school students are included in Figures 3 through 5.
ABOUT THE AUTHOR
Dr. Tom Rochon
Dr. Thomas R. Rochon was the president of ERB from 2017-2025. His previous career in higher education included faculty appointments at Princeton University and Claremont Graduate University, a postdoctoral fellowship at Stanford University, and a year as a Fulbright Scholar in Japan. Administratively, Tom has served as dean and provost at Claremont Graduate University in California, provost at the University of St. Thomas in Minnesota, and president of Ithaca College in New York.
Within the world of educational assessment, Tom previously served as executive director of the GRE testing program at the Educational Testing Service (ETS). Tom’s scholarly books and articles include Mobilizing for Peace: The Antinuclear Movements in Western Europe (Princeton University Press, 1988) and Culture Moves: Ideas, Activism and Changing Values (Princeton University Press, 1998). He received his bachelor’s degree, master’s degree, and Ph.D. all in political science from the University of Michigan.
ABOUT THE AUTHOR
Aaron Shuman
Aaron Shuman is a technology professional with 25 years of experience in data-driven design and implementation. He has an educational background in statistics, psychology, and computer science with degrees from Yale University. In his current role at ERB, Shuman is the Director of Analytics and Insights, with a focus on providing ERB members with a better understanding of their students’ performance over time. His specific responsibilities include data integration and analysis for 360 Access, ERB’s data reporting platform.
