By Adrienne Hu and Glenn Milewski
For the Spring 2023 reporting cycle, the rolling norm continues to use four consecutive years of data.
CTP norms are typically updated annually for each reporting cycle (fall and spring). These “rolling norms” as they are often called, are obtained after gathering three years of data, and in the fourth year, deleting the earliest year and adding data from the most recent administration. During the pandemic, however, when many member schools suspended in-person testing, ERB “froze” the CTP norms. Instead of updating them with limited data from that unusual moment in history, ERB provided members with the rolling norms established immediately prior to the pandemic.
Starting with the spring 2022 reporting cycle, ERB “unfroze” the rolling norm and updated it to be based on four years of data instead of three. This approach was used so that the norms were based on a mix of pre-pandemic and pandemic data. For the spring 2023 reporting cycle, ERB is once again updating norms and continuing to use four consecutive years of data in the rolling norm. Figure 1 shows that the norms for the spring 2023 reporting cycle are based on test-takers from 2019 through 2022, although test-taker volume was extremely reduced in spring 2020.
When comparing the spring 2023 rolling norms to the spring 2022 rolling norms based on independent school students, we found that the rolling mean continued to decrease for certain subtests. When we further examined this phenomenon, we discovered that for certain subtests, scores in 2021 and 2022 were consistently lower than the pre-pandemic scores, resulting in an overall lower rolling norm in 2023 as compared to 2022. This effect appeared to impact more notably on the Mathematics subtest, the Quantitative Reasoning subtest, and certain ELA subtests in the middle and upper levels. In Figure 2 below, average CTP scores for the Mathematics and Quantitative Reasoning subtests in grades 3 through 10 are shown for the past four spring testing seasons. The sparklines in Figure 2 demonstrate visually the trend in performance across years. In particular, it’s clear that in Grade 8 Mathematics, scores went down by 14 points from 2019 to 2021 and remained flat in 2022. Those who still tested in spring 2020—when suspended testing was at its peak—demonstrated better performance with a higher mean scale score on the Grade 8 Mathematics subtest. But with the test volume so low in that testing window, the performance of this particular group has minimal weight on the rolling mean.
*The 2020 means are from a very restricted test-taker population in 2020 Spring and should be interpreted with caution.
What does this mean in terms of percentiles for the member schools? It means that schools should expect to see a slightly higher percentile rank for a student with the same scale score in either Mathematics or Quantitative Reasoning as compared to last year. For example, in the updated norms of Spring 2023, a Grade 8 student who has a scale score of 750 in Mathematics will have a percentile of 74. In the previous Spring 2022 norms, a Grade 8 student with the same scale score would have a percentile of 70.
These findings are partially consistent with the growth analysis that Rochon and Shuman (2021) did based on a cohort testing over two CTP fall windows. They found that Quantitative Reasoning was one of the academic disciplines that was more impacted by COVID. In a subsequent study, Rochon and Shuman (2022) then found a restoration of pre-COVID levels of learning growth during 2021. For the highest-performing students, there was even an acceleration of growth compared to their pre-COVID yearly growth rates. However, the learning impact of COVID means these growth rates are layered on top of a lower level of absolute performance. Accelerated growth will need to continue for several more years for norms to reach their pre-COVID levels.
Moving forward, ERB will continue to use a four-year rolling norm. This means that in spring 2024, the rolling norm we will be using will not include data from before the pandemic. It is too early to say in what direction the average scores for the 2024 rolling norm will go, as this depends quite heavily on spring 2023 test-taking data. We’ll keep you posted as we learn more!
To learn more, join our session ERB Statistics and Student Growth and Learning at the ERB Conference!
