Learning Growth Among Students in ISAS Schools Before, During, and Since the COVID-19 Pandemic

Executive Summary

The impact of the COVID-19 pandemic on student learning was profound in all schools— public and private, secular and faith-based. The COVID impact was especially great on those students whose academic performance was relatively weak before the pandemic since those are the students who most rely on teachers and other educational supports. 

Among ISAS schools that are also members of ERB, however, student learning has made a complete recovery from the COVID disruption. Assessment data from the 2023-2024 school year show that the pace of student learning fully matches, and in mathematics even exceeds, pre-pandemic rates of growth. These data demonstrate that concerns over long-term COVID impacts on student learning do not apply to students in ISAS schools. 

Introduction

COVID-related disruption of normal social interaction in March 2020 will long remain seared in our collective memory. Workplaces were abandoned, hospitals and senior centers could not accept visitors, and schools closed their doors—all on short notice and with little idea how long those closures might last. While many areas of endeavor have since returned to normal, concerns linger over the long-term impact of the COVID disruption on students in their formative years. A National Assessment of Educational Progress (NAEP) report published in 2022 showed that:  

“…Nine-year-old students, typically in fifth grade, experienced the largest-ever score decline in reading and the first-ever recorded score decline in math. Peggy Carr, Commissioner of the National Center for Education Statistics (NCES), was cited in USA Today as saying ‘It’s clear that COVID-19 shocked American education and stunted the growth of this age group of children.’”¹ 

Commentary on COVID’s impact on education has been dominated by test results from students in public schools. In a series of studies dating back to late 2020, however, ERB has mined its database of Comprehensive Testing Program (CTP) results to understand the extent to which COVID-related trends affected student learning in independent and faith-based private schools. 

The purpose of this paper is to summarize our findings on the COVID disruption among ERB members in the first 18 months of the pandemic. We will then contrast that disruption with the striking restoration of student learning rates among ISAS schools beginning in the 2021-2022 school year and continuing to the present. The story told by our student learning data is one of effective adaptation and continued educational excellence displayed by administrators, teachers, and students alike.   

The Initial Impact of COVID-19 on Student Learning

To assess the impact of Spring 2020 school closures on student learning, we analyzed results from the Comprehensive Testing Program (CTP) from a panel of students who took the test battery in Fall 2018, Fall 2019, and Fall 2020. Using data from all ERB member schools, we compared the test score growth rate from Fall 2018 to Fall 2019 (pre-COVID) with the growth rate from Fall 2019 to Fall 2020 (spanning the school closures in Spring 2020).²

Figure 1 demonstrates the impact of the transition to remote education on student learning. Data from Grades 2 through 7 are combined in this graph, though inspection of the results for each grade shows a great deal of consistency.

Figure 1

Year-Over-Year Learning Gain Among Independent School Students

COVID-related disruptions can be clearly seen to have reduced the rate of student learning during the Spring and Fall of 2020. Student growth from 2019 to 2020 in Mathematics, Reading Comprehension, and Writing Concepts & Skills was approximately 65 to 70% of the amount of growth exhibited by those same students the previous year. While significant, it is worth noting that a 30% loss of growth momentum was far less dire than results reported at the time based on national studies of statewide testing in public schools. 

Figure 1 also shows that the impact on learning growth was greater in Quantitative Reasoning and Verbal Reasoning than in the CTP subject tests. Students had roughly half the amount of growth in reasoning skills from Fall 2019 to Fall 2020, compared to the growth shown by those same students one year earlier.

Reasoning skills are built over time by acquiring a depth of understanding that enables students to connect new material to what they already know and to work through problems they have not yet encountered. That kind of deep engagement with the material—mastery rather than surface understanding of the facts—was disproportionately impacted when students abruptly lost daily contact with teachers and peers in the classroom. 

These findings raise the question of whether the COVID disruption was greater on some groups of students than others. To address this, we examined the rate of learning growth among the strongest and weakest students by dividing our population into the bottom 25% in test performance (“Low”), the middle 50% (“Middle”), and the top 25% (“High”) based on their test scores one year earlier, in 2017-2018.

Figure 2

Test Score Growth by Level of Student Performance

Between Fall 2018 and Fall 2019, students in the Low group showed the greatest degree of test score gain (14.6%), followed by the Middle group (9.0%) and the High group (6.5%). This is the usual pattern of growth from one year to the next. Some students in the low-performing group one year are ready to accelerate their growth, aided by the fact that teachers and learning specialists are focused on their needs. Students in the Low group also have more opportunities for growth since there is more material still to be learned at their grade level.³ 

In the pandemic-disrupted Spring and Fall of 2020, however, this typical pattern of growth changed dramatically. The amount of test score growth over that period was almost indistinguishable among the three groups, ranging from 6.3% in the lowest group to 5.9% in the highest group. While consistency in learning growth across student performance levels may at first seem gratifying, it is important to note that the lowest-performing student group was actually the most disadvantaged compared to the growth they would typically have experienced from one year to the next. The virtual and hybrid learning models that schools were forced to adopt on short notice thus had the greatest negative impact on those students who were already in the lowest achievement group. 

This pattern is exaggerated in the verbal and quantitative reasoning tests, with top-quartile students demonstrating slightly more learning growth than those in the bottom quartile. The strongest students seem not to rely as much on teacher interaction to incorporate new knowledge into their reasoning process, thus enhancing their problem-solving success. 

These findings outline the contours of the challenge faced by teachers and administrators as the COVID pandemic subsided and school attendance returned to normal. The central role of teacher-student interaction in the classroom needed to be re-established, with a particular emphasis on supporting the lowest achieving students and on supporting the development of reasoning skills.

Restoration of Student Learning Patterns Among ISAS Schools Since the COVID Disruption

We have now had three full academic years relatively free of COVID disruption. To see how students have responded, we compare student growth in ISAS member schools during the final pre-COVID year of 2018-2019 with post-COVID growth rates from 2021-2024. 

Figure 3 shows the results of five CTP tests grouped in two different ways, contrasting Math tests with English Language Arts tests, and then regrouping those same tests to contrast subject tests with assessments of reasoning.   

Figure 3

Comparing Test Score Growth Pre- and Post-COVID

Using the 2018-2019 learning growth results as our baseline for comparison, we can see that student growth in 2021-2022 was actually a bit higher than before COVID, particularly in ELA. This is due to a “catch-up effect” in which students rebounded from the lower growth rates characteristic of the 18 months of COVID disruption. In Mathematics, that rebound was not temporary, as reflected in the continued acceleration of Math learning growth during the two most recent school years. 

In short, student learning has made a substantial recovery from the COVID disruption among ISAS schools and, in some areas, has even outpaced pre-COVID growth rates. This is a remarkable outcome that speaks volumes about the resilience of the learning environment in ISAS schools.  

Remembering that reasoning skills were especially hard hit during the COVID disruption, we also grouped CTP tests into those focused primarily on subject knowledge—Math, Reading Comprehension, and Writing Concepts and Skills—and those focused on ability to use subject knowledge to solve multi-step problems (Quantitative Reasoning and Verbal Reasoning). 

In 2021-2022, the effects of COVID were still apparent in the pattern of student learning growth, as seen both in the catch-up acceleration in the subject tests and in the lingering effects of the COVID-induced slowing of reasoning growth. In the two most recent years, student learning growth in ISAS schools exceeded pre-COVID rates both in tests of subject knowledge and reasoning ability. These results stand in sharp contrast to reports on student learning based on state and national test data from public school systems.  

How successful have ISAS member schools
been in supporting the lower-achieving students
who were most impacted by school closures
during the first phase of the pandemic?

Figure 4 shows a re-establishment of the stair-step pattern in which lower-achieving students experience higher rates of learning growth. In 2021-2022, we saw only a partial restoration of that pattern, as lower-achieving students had at that point not yet recovered their pre-COVID rate of learning growth in either Math or ELA. By 2023-2024, however, learning growth among students in the bottom quartile of performance has fully recovered pre-pandemic levels. Meanwhile, high-achieving students have actually accelerated their learning compared to pre-pandemic rates.

Figure 4

Test Score Growth by Level of Performance

Figure 5 contrasts growth of subject knowledge and reasoning skills among students in the bottom quartile of performance, those in the middle two quartiles, and those in the top quartile. Bottom-quartile students have fully recovered their pre-COVID rates of growth in both subject and reasoning tests. Students in the middle 50% and those in the top quartile have also fully recovered pre-COVID growth rates, with high-achieving students exceeding pre-COVID learning growth rates in both subject knowledge and tests of reasoning skills.  

Figure 5

Test Score Growth by Level of Performance

Two factors may be contributing to the higher rates of learning growth by students in the top quartile of achievement. One is that these students have a catch-up opportunity created by the initial COVID impact. High-achieving students are usually near the ceiling in growth potential because of their existing mastery of grade-level material. With the impact on learning experienced during 2020-2021, however, there was an increased opportunity to accelerate subsequent learning growth. 

It is also possible that the most capable students flourished precisely because of the habits of greater independence required of them during the initial phase of the pandemic. The 2022 NAEP study of academic performance among 9-year-old students nationwide was paired with a survey in which they were asked a number of questions about their educational experience during COVID. The NAEP report concludes that:  

“Higher-performing students reported more confidence in their ability to recognize when they don’t understand something they are learning, ask for help when they need it, and find learning resources online to learn more about something they don’t understand, compared to their lower-performing peers.”

The COVID-19 pandemic taught us many things about work, community, and human nature. One of those lessons specifically applicable to education may be the extent to which the most capable students flourish when given greater independence and responsibility for their learning. It is to the great credit of ISAS schools that their strongest students have continued an accelerated rate of learning in the years since the pandemic ended.  

Conclusion and Action Implications

These findings on student learning growth suggest a number of takeaway lessons. First and most importantly, ISAS member schools have been highly successful in re-establishing pre-COVID rates of learning growth. The initial COVID impact on learning growth was greater among weaker students and greater in the areas of Verbal and Quantitative Reasoning than in the acquisition of subject knowledge. Assessment results from the last two years show that these impacts of COVID disruption on learning growth have been completely erased. 

The accelerated learning growth experience of the highest-performing students during the last three years presents ISAS schools with an opportunity to reflect on ways they might capitalize on the habits of independent learning these students have developed. The goal here would be to identify the strategies that have enabled the strongest students to achieve unprecedented rates of learning growth and to adapt those strategies to benefit all students. 

COVID-19 created a disruption in school routines that was without precedent in living memory. ERB’s assessment data make clear that ISAS schools can take pride in their effectiveness in sustaining an environment of educational excellence despite the challenges of the pandemic. 

---

¹ USA Today, September 1, 2022. For the NAEP results, see https://www.nationsreportcard.gov/highlights/ltt/2022

² Nearly 2 million subject-specific CTP tests are typically taken each year by over 250,000 students drawn from over 1,000 schools, making any analysis of learning growth trends highly reliable. During fall 2020, however, fewer students took CTP tests than usual. Those who did so often took the tests at home, delivered online with their teachers serving as remote proctors. To ensure comparability, we examined score trends from 2018 and 2019 for this much smaller sample and found that they are representative of the larger universe of CTP test takers.

³ This is an example of what statisticians call “regression to the mean”—a tendency of those who are low in one measurement to be higher in a subsequent measurement, while those who are highest in the first measurement will tend to be less high in the next. We find this consistently across each of the CTP subtests.