Growth Model 101
What is the CORE Academic Growth Model?
The CORE Academic Growth Model aims to measure the impact of schools on student academic progress over time. The model uses a statistical technique to identify the effects of the education system on student achievement, isolating these effects as much as possible from other factors that influence student achievement such as prior student knowledge. The education system includes all elements of a school environment that contribute to student learning, such as principals, classroom teachers, and support teachers.
Why is measuring growth important?
The CORE Academic Growth Model is a key element of CORE’s School Quality Improvement Index (SQII). One of the simplest ways school effectiveness has been evaluated in the past has been to use student achievement data, such as proficiency rates in Mathematics and English Language Arts. While useful and important, achievement data alone do not provide a complete picture of how students are performing and how schools and districts are doing in improving student learning over time. The CORE Academic Growth Model helps to determine how much growth students have made from one year to the next, and most importantly, allows districts and schools to determine their impact on the academic growth of students.
The CORE Academic Growth Model measures the school system’s effect on learning in that year, adjusting for prior knowledge and other student characteristics which may influence student growth. It ensures that schools serving students with different prior achievement levels and characteristics have an equal chance of showing high growth.
Which students count in the CORE Academic Growth Model?
Results will be provided for grades 4-8 in Mathematics and English Language Arts. In order to count in the CORE Academic Growth Model calculation, a student must:
● Have at least two years of valid test results to measure growth
Models at the High School level are being investigated for feasibility in the CST-to-SBAC transition period and beyond. To date, we have provided participating districts with growth model that utilizes 2015 CAHSEE performance as the prior achievement data for 11th grade students’ 2016 SBAC performance. We are also exploring a method that will utilize 8th grade performance as the predictors for 11th grade performance in a high school academic growth model.
Why does the CORE Academic Growth Model only include Mathematics and ELA?
In order to produce the CORE Academic Growth Model, standardized test scores must be available for students from one year to the next. Tests in Mathematics and English Language Arts are most suitable because students take those tests in successive years. For example, a student’s Spring 3r d grade and Spring 4t h grade ELA performance can be used to measure ELA growth during the 4t h grade school year.
In some cases, where consecutive test years are not available or will not be available, such as in high school, the CORE Academic Growth Model may be able to measures growth of students from earlier grades to the present. Feasibility of such models are still being investigated.
How is growth aggregated from the student level to schools?
In the growth results, each student’s growth is compared to the growth of students with similar test histories and characteristics from across the CORE Districts. The actual growth of a particular student could be slower than, the same as, or faster than the growth of similar students.
The CORE Academic Growth Model aggregates these student results to determine the contribution of a school to students’ growth in achievement. When doing this, the model considers by how many points each student did not meet or exceeded average growth for similar students. For example, a student who exceeded average growth by 5 points will count more positively toward his or her school’s rating than would a student who only exceeded average growth by 1 point.
How do I interpret the CORE Academic Growth Model results?
The result of the CORE Academic Growth Model can be interpreted as a measure of a school’s contribution to a student’s growth in achievement from one year to the next.
Results are reported on a 0-100 Student Growth Percentile (SGP) scale where higher numbers mean higher growth and lower numbers mean lower growth.
On this scale “50” represents students growing at an average rate compared to students with similar test histories and characteristics from across the CORE Districts. If a school receives a “50”, that means their contribution to students’ growth in achievement is close to the CORE Districts average.
Numbers higher than “50” represent students growing at a rate faster than average. These schools’ contributions to students’ growth in achievement is higher than the CORE Districts average.
Numbers lower than “50” represent student growing at a rate slower than average. These schools’ contributions to students’ growth in achievement is lower than the CORE Districts average. A result less than “50” does not mean that students lost knowledge, it means that they are growing at a rate slower than average for similar students across the CORE Districts.
What do the numbers in the subgroup level results represent?
The subgroup level results are the percentile rank of the growth of students within that subgroup relative to all students in CORE. To calculate this number, the growth scores for each individual student are averaged by subgroup. To account for the smaller sample sizes within subgroups, these averages are combined with the school averages using a statistical methodology called shrinkage to produce growth estimates. These subgroup growth estimates are then percentile ranked based on where the estimate would fall within the normal distribution of all CORE students (as opposed to calculating the percentile rank empirically). Finally, results with 20 or fewer students are suppressed and these are the reported results.
What does the CORE Academic Growth Model adjust for?
In order to create equitable measures of schools contribution to student growth, the growth models used must be designed to adjust for the context in which schools operate. However, there is no straightforward answer as to what factors should be included in the “right” model; the CORE Districts have considered questions of fairness and accuracy in making these decisions while continuing to set high expectations for all students. To account for differences in student context, the CORE Academic Growth Model adjusts for the following factors at the individual student and school average levels:
- Prior test history (both Math and ELA)
- Economic disadvantaged status
- Disability status (and severity level)
- English learner status
- Homelessness status
- Foster status
How accurate is this method?
The CORE Districts’ approach to building our growth methodology draws upon the expertise of advanced analytic providers; a technical advisory council of growth experts; and school and district leaders to ensure that the decisions made are the best they can be. The goal of this process is to build a growth methodology that is technically accurate and aligned to the CORE Districts’ policy goals.
For the portion of the School Quality Improvement Index that does use the CORE Academic Growth Model results, many techniques are in use to make those results as fair and accurate as possible. For example, to account for inherent limitations in standardized test scores, the growth model employs a correction for “measurement error” in prior test scores. Additionally, to guard against the influence of random events affecting the results, the growth model is based on schools’ impact on groups of students over multiple grade levels. In this way, individual student factors such as student effort or a “bad test day” are evened out across the group. Several other techniques are also applied to acknowledge and account for other limitations to produce the most accurate results possible.
All methods for measuring student achievement and growth have some amount of inherent error. For this reason, the School Quality Improvement Index does not rely solely on the CORE Academic Growth Model outcome and rather employs multiple measures to assess student learning and schools’ contribution.
Does the model set different expectations for students?
The CORE Districts recognize that students start at different places, and want to identify and foster growth for all students. In achievement models, high achievement results are often more indicative of students’ background characteristics than the effectiveness of teaching within an education system. Academic growth is more appropriate for measuring effectiveness of instruction rather than looking only at end-of-year performance.
In order to accurately measure a school’s impact on student growth, the model adjusts for external factors beyond the scope of schools in creating growth predictions. At the same time, these “predictions” do not mean setting different expectations for different students at the beginning of a year. Instead, the predictions are created using a statistical model at the end of the school year and actual student progress is compared to a student’s predicted performance.
The CORE Districts continue to have high expectations of growth for all students and the goal of the CORE Academic Growth Model is to create accurate measurement of growth to support those expectations.
Is there a disadvantage for schools with high achieving students in the CORE Academic Growth Model?
The CORE Academic Growth Model results will adjust for the prior test scores of students to be a fair and accurate measure of growth for students of all proficiency levels. Rather than assume that students all along the test scale will make the same amount of growth in test score, the CORE Academic Growth Model adjusts for growth trends at different points in the test scale. For example, the test itself may have properties that would cause a trend where higher achieving students would gain fewer points on the test compared to lower achieving students. If this is found to be the case, the growth model results will adjust for that effect. In general, the growth model compares the growth of a student to the growth of similarly achieving students.
When does the analysis occur to produce the student growth model results?
The CORE Academic Growth Model uses retrospective growth analysis. This means that the growth model and relevant statistical predictions for student growth are not calculated until all assessments have been scored and are available from the growth period in question (e.g., the 2015-2016 growth period cannot be scored until the spring 2016 assessment data are available).
How is CORE’s Academic Growth Model different from how California measures change in status for the Local Control Accountability Plan (LCAP) Rubrics?
The CORE Academic Growth Model measures the longitudinal growth of individual students over time and adjusts for student and school context. It is designed to measure the impact of educators on student growth.
The LCAP rubrics measure the achievement (status) of students at a point in time and compare the achievement (status) of different students over time by comparing the achievement of students in the current year to a groups of students at the same school at the prior year.
I thought the new state tests (aka, SBAC, CAASP) have a “vertical” scale, so why don’t we just measure growth by the change in each student’s scale score?
One challenge to relying upon a vertical scale of an assessment is the difficulty in accurately creating a test scale in such a way that scale score growth is comparable across starting points, across grades, and across years. Preliminary analysis of the SBAC test scale has shown that growth at different grade levels is not comparable, and thus drawing conclusions about school impact is not straightforward for schools serving different grade configurations. For example, students at the earlier grade levels tended to grow almost twice as many points on the SBAC from year-to-year as did students in the upper grade levels.
Secondly, even if we found that on average, students grew roughly the same number of points on the vertical scale at different starting points, across grades, and across years that does not mean that the average growth in a school is an apples-to-apples measure of that school’s impact on student academic growth. The additional context adjustments that the CORE Academic Growth Model employs are designed to compare the growth of a school’s students to the growth of similar students from across the CORE districts. In this way, the results of the CORE Academic Growth Model are a more equitable measure of educator impact than would be the results of a model that relies solely on the vertical scale of an assessment.
Growth Model Confidence Interval Analysis
How are results reported in the CORE Growth Model?
The results include:
- The point estimate, the best estimate of a school’s impact on student academic growth
- A confidence interval, a way to represent the reliability of the estimate
Does a wide confidence interval mean the estimate is not useful?
No, it is simply truth in advertising on what we can measure and is a scientific representation of the range which we can be relatively sure the impact of the school lies. The best estimate of the school’s impact is the point estimate.
What does the confidence interval represent in the CORE Growth Model?
The CORE Growth Model is a statistical model that is designed to estimate the impact of school teams on the academic growth of students. It accomplishes this by pooling data from across the CORE Districts (such as student test scores, student-level demographics, school-level demographics) to compare the actual growth of students in a particular school to the growth of similar students from across CORE. If a group of students grew faster (or slower) than similar students, the result can be interpreted as the school teams’ impact on student academic growth.
The model produces a single value, the point estimate, that represents the best statistical estimate of educators’ impact on students. The point estimate is translated to a 0-100 percentile scale for reporting. However, no educational measure or statistical model is perfect, and a typical way to represent the reliability of the results in a confidence interval, which is why, for instance, student test scores now show a confidence interval in reports to students and parents by the state of California. Narrower confidence intervals represent higher reliability, while wider confidence intervals represent lower reliability.
The confidence intervals reported for the CORE Growth Model are 95% confidence intervals. They represent a range of plausible values for the indicator the model is attempting to measure – the impact of school teams on student academic growth. When interpreting the results of the analysis, the most likely true value of impact is the point estimate and the likelihood diminishes farther away from the point estimate in both directions.
Why are some confidence intervals larger than others?
There are a number of factors related to confidence interval size. One of the key driving factors is the amount and quality of data driving the analysis. For example, when estimating the true impact of a school, more reliable estimates can be produced when there are data from more students to analyze – a results based on the growth outcomes of 200 students will tend to have a tighter confidence interval than a result based on the growth of only 20 students.
In the CORE Growth Model, there is also another major factor driving confidence interval size – the translation of results to a percentile scale. The CORE Districts decided to translate growth results into a scale from 0 to 100 to put an approximately equal number of schools into each of the 1-10 SQII categories in order to improve consistency in how to interpret growth results. All CORE growth model results are put into the scale below.
This rule was chosen to aid in comparison of results in different grades and subjects. Since results are equally distributed in each phase of the analysis, the interpretation of each growth level remains consistent. For example, earning a “Level 7” means the result is between the 60th and 69th percentile of impact for each growth model result (4th grade Mathematics, 8th grade ELA, etc.)
As a consequence of this decision, there is more certainty in the results of schools that are far above or far below average. There is less certainty in the results of schools that are near the middle of the scale.
This effect can also be seen when examining the relationship between the confidence interval size and the value of each school’s point estimate. Schools that have a point estimate near average tend to have confidence intervals that are reported to be much wider percentile ranges even if the actual confidence interval size may be similar. This is an artifact of the reporting of results on the percentile range scale.
