To access Quick Links, visit our text-only version.

. Public Schools of North Carolina . . State Board of Education . . Department Of Public Instruction .

2000-01 ABC PROGRAM INFORMATION

THE EOC PREDICTION FORMULAS

Revised February 6, 2001
Vol. 1, no. 5

The use of end-of-course prediction formulas to determine growth/gain on the ten multiple choice End-of-Course (EOC) tests is a recently approved modification to the ABCs. The formulas, designed for use with the EOC, were approved at the May 2000 State Board of Education (SBE) meeting for implementation beginning in the 2000-01 school year. In January 2001 the SBE approved fine-tuning of the EOC prediction formulas and weighting the ABCs growth/gain composites for the 2000-01 school year.


Why change the way gain is computed for the high school EOC tests?

The ABCs of Public Education, first implemented in the 1996-97 school year, measured student growth by grade level and subject by comparing the end-of-grade (EOG) test results from one year to the next, except in the case of grade 3, where a pretest was administered during the same school year. High schools expressed concerns about their participation in the ABCs, noting that there was no comparable mechanism in place at the high school that could measure developmental growth across a course or subject. Although attempts were made to develop a pretest to posttest model at the high school, such a model did not prove feasible. Continuing concerns about the ABCs at the high school level contributed to the postponement of implementation until the following year. During this time a Steering Committee for Assessment and Accountability (appointed by the SBE) collected input and studied the issue of high school accountability.

In 1997-98, a high school model based on an EOC index was ready for implementation. This model based EOC gain on an index that measured current year gains in EOC scores over two-year historical baselines. Thus in the high school model, schools were not held accountable for change, or improvement, in the same group of students over time, but rather the gain of one group of students over previous groups of students. There was strong sentiment among stakeholders that a cohort model based on the same students' growth over time should become a part of the high school model. Research and planning in this direction continued, and one step was the development of the North Carolina High School Comprehensive Test (NCHSCT). This test was designed to measure growth in reading and mathematics from the time of the EOG administration at the end of grade 8 to grade 10; it was administered in 1997-98 to collect data for further analyses.

In 1998-99, a growth component based on the NCHSCT was added to the high school model. Growth from grade 8 to grade 10 in Reading and Mathematics would be measured using the pretest to posttest model. With this addition, the high school model had at least one component that was based on the same students' growth over time. The other five EOC tests continued to provide results for the index computations, while work towards a prediction model continued. In January 2001 the SBE approved fine-tuning of the EOC prediction formulas for the 2000-01 school year. These refinements include: 1) adjusting EOC prediction formulas exemplary growth/gain standard; and 2) weighting the ABCs growth/gain composites.

Various statistical models of an EOC prediction formula were investigated. The formulas that were tested were measured against three criteria: predictability, simplicity and equity. After two years of research and analyses, the newly approved EOC prediction formulas met these criteria.


Are the EOC prediction formulas applied only to high school EOC courses?

No. The formulas are used to compute growth/gain wherever the EOC courses are offered, regardless of the grade configuration of the school. For example, a K-8 school that offers Algebra I to eighth graders would use the formulas (with seventh grade EOG math as predictor) to compute growth/gain in that subject; the growth/gain would be included in the composite growth/gain for the school.


How do calculations with the new prediction formulas differ from calculations using the index?

In the EOC prediction formulas, an equation is used to calculate an ABCs goal (or expected score) for each school on each EOC test. Each expected score is based on the proficiency of the students when they were in previous grades or courses. Proficiency is determined by students' performance (scores) on the North Carolina End-of-Grade (EOG) or End-of-Course (EOC) tests, which serve as predictors of the same students' performance in the EOC course where they are currently enrolled. The EOG or EOC test(s) serving as predictor(s) are different for each EOC test based on the criteria of predictability, simplicity, and equity.

Let's look at an example of a calculation to find a school's ABCs goal (expected score) for Algebra I. The equation used to compute the target score for Algebra I is

  • Algebra I Expected Score = b0 + (bIMP x IMP), where
    b0 is the state average performance for the EOC = 60.4;
    bIMP is the value used to estimate the effect of the school's average math proficiency on the predicted average EOC test score = 0.88; and
  • IMP is the index of mathematics proficiency equals [school's average EOG Grade 8 Math scale score minus 176.1,(the state's average scale score)].

Substituting the values from the prediction formula parameters for End-of-Course performance, the equation looks like this:

  • Algebra I Expected Score (Predicted Algebra I Mean) = 60.4 + [0.88 x (Math ­ 176.1)]

Step 1: Identify a group of students currently enrolled in Algebra I with predictor scores. In this case, the group of students must have scores on EOG Math from grade 8.

  • Important considerations in selecting a matched set of students: Some students currently enrolled in Algebra I may not have a grade 8 EOG score in math, for a number of reasons. There may be students who have transferred from other states; a student may have been absent and failed to complete a make-up. Enrollment can change daily due to the addition of new students, transfers, and withdrawals. This makes it impossible to compute a totally accurate expected score until the current enrollment is "captured" on the first day of EOC testing.

Step 2: Find the average EOG Math (grade 8) score for the matched group of students; using this average in the equation, determine the expected Algebra I score. Let's say the average EOG Math score in grade 8 for this group was 178.

  • Algebra I Expected Score = 60.4 + [0.88 x (178 ­ 176.1)]
    Algebra I Expected Score = 60.4 + 1.672= 62.042*, or, 62.0 (rounded).

    This means that to reach the expected score for expected growth/gain, the school must have an average Algebra I EOC score that equals or exceeds 62.0.

    *This sum reflects full precision carried throughout each computation, and does not reflect the sum of the rounded numbers shown in this example.


How is gain computed?

The ABCs accountability model was implemented on the assumption that the number of students in a school would be roughly equivalent from grade-to-grade. Every year since the implementation of the ABCs a small number of students in a subject or grade have had a positive or negative influence on the ABCs growth/gain of a school. Weighting of the ABCs growth gain composites was adopted to deal with such inconsistencies. Gain occurs when the actual EOC average score for a matched group of students in a school is equal to or greater than the predicted EOC target score, or ABCs goal for the school. The gain is computed much like growth is computed with the EOG tests. The school's expected score (ABCs goal) is subtracted from the actual EOC average of the group; the difference is divided by the standard deviation of differences (for schools in North Carolina), and the quotient is called the "standard EOC gain" in a given subject. The standard expected growth/gain is then multiplied by the number of scores for Algebra I and is then divided by the total number of scores for all ABCs growth/gain components. This yields a weighted standard expected growth/gain.

Let's continue with the Algebra I example from above to illustrate. The group's actual mean score for the current year is 63. The expected score computed for Algebra I was 62.0.

These values are placed in a worksheet and the expected growth/gain for Algebra I is computed as follows:

 Components
A
B
C
E
F
G
H
I
Actual EOC Average
Predicted EOC Average
Actual Minus Predicted
     
Standard Deviation Of Differences*
Standard Expected Growth/Gain
Algebra I 63.0 62.0 1.0       3.3 +0.3**

 

*Standard Deviations of Differences are the same for both expected and exemplary gain calculations for a given EOC course, EXCEPT English II (where gain is based on the index model). See page 7, column G.

**Full precision, though not shown here, is used in all ABC Tools calculations; the final composite is rounded to tenths.

In a high school, the weighted standard expected growth/gain for each of the ten multiple-choice EOCs is added to the other growth/gain components. Those components are:

  • standard expected growth in Reading and Mathematics from grades 8 to10;
  • gain in the competency passing rate;
  • gain in the College/University Prep/College Tech Prep component;
  • index gain in English II, and
  • change in the ABCs dropout rate.


What about exemplary growth/gain?

For exemplary gain, the same EOC prediction formulas are used. However, the state average performance is multiplied by 1.03. This means that the exemplary standard is approximately 3% greater than the expected EOC expected score. The formula for exemplary growth/gain in Algebra I is:

  • Expected Score for Algebra I Exemplary Growth/Gain = (b0 x 1.03) + (bIMP x IMP)

From the earlier example for Algebra I, computations for determining exemplary growth/gain would follow the two steps below.

  • Step 1: Multiply b0 x 1.03.
    60.4 x1.03 = 62.21

  • Step 2: Substitute the appropriate values in the formula and complete the calculations.
    62.21 + [0.88 x (178 ­ 176.1)]
    62.21 + 1.672 = 63.88*, or 63.9 (rounded).

This means that to reach the exemplary expected score, the school's average performance on Algebra I EOC tests must equal or exceed 63.9

*This sum reflects full precision carried throughout each computation, and does not reflect the sum of the rounded numbers shown in this example.  


What impact will the EOC prediction formula have on a school's ABCs status?

Implementation of the EOC prediction formulas in the ABCs calculations will essentially "raise the bar" for high schools. This is because the formulas are based on a more recent population of students than was available when the index model was developed. Weighting the ABCs growth/gain composites will eliminate the concern over small groups of students having the same impact as large groups of students in determining whether a school has met growth standards.    


Are there any other implications?

Since computations rely on previous scores for a group of students, there may be fewer students to include in the calculations. Statewide analyses of 1998-99 student test data indicated that nearly 90 percent of the students in North Carolina had the necessary data for using the EOC prediction formulas. It is critical that previous test data for students be included in the SIMS or NCWISE systems. School administrators interested in improving the EOC average score in their school should focus on the achievement of all students; not just students who score at levels I and II. It is imperative for teachers to teach the Standard of Course and monitor student work routinely.  


Are there time requirements like the EOG 91-day rule for prediction?

No, there is no minimum number of days in membership requirement for use with the EOC prediction formulas. It is not currently feasible to collect data on days of membership in EOC test courses. Keep in mind, however, that the growth computations for reading and mathematics from grade 8 to grade 10, which are based on the North Carolina High School Comprehensive Test, require 160 days in membership.


Are there data requirements for using the prediction formulas?

No, there is no minimum number of scores for each course in order to use the prediction formulas. However, there must be at least 30 scores contributing to the school's growth/gain composite for the scores to be included in the ABCs.


Do testing requirements (95% rule, excessive exemptions) still apply?

Yes. High schools that test fewer than 95 percent of students subject to EOC tests and the NC Comprehensive Test may not receive recognition, rewards, and/or incentives. In addition, the 98 percent rule applies to schools that have grades that fall within the K-8 range; schools that test fewer than 98 percent of eligible students may not receive recognition, rewards, or incentives. A school comprised of grades 6 through 12, for example, must meet the 98 percent rule in grades 6 through 8, and the 95 percent rule in the high school grades. Schools that violate the testing requirements for two consecutive years may be designated as low-performing by the State Board of Education. Please note that the only allowable exemptions involve LEP students who have been enrolled in school for less than two years.


Worksheet for Computing Weighted Expected Growth/Gain Composite for a High School Starting with the 2000-2001 Accountability Cycle

A
B
C
D
E
F
G
H
I
J
K
Components
Actual EOC Average
Predicted EOC Average
Actual minus Predicted
Standard Deviation of Differences
Standard Expected Growth/ Gain
# (n)
Weight (n) divided by (N)
Weighted Growth/ Gain (H x J)
Algebra I             3.3        
Biology             2.6        
ELPS             3.1        
English I             1.8        
US History             2.2        
Algebra II             2.9        
Chemistry             2.5        
Physics             3.3        
Physical Science             2.5        
Geometry             2.5        
  Current Index Year Two Index Year One Index Baseline Difference Subtract 0.1 Standard Deviation of Expected Gain1        
English II             7.6        
  Current % Year Two % Year One % Baseline Difference Subtract 0.1 Standard Deviation        
CUP/CTP             10.0        
Comp-Test (Gr. 10) Actual Growth Expected Growth Actual Minus Expected       Standard Deviation        
Reading             1.6        
Math             2.0        
  Grade 10 Passing Rate % Grade 8 Passing Rate %     Difference Subtract 0.1 Standard Deviation        
Competency     Year One(%) Year Two (%)     12.8        
Dropout2             2.0        
  Total # (N)      
 
Weighted Growth /Gain Composite  
1Calculations for English II are based on the index model, so the Standard Deviations differ for Expected and Exemplary Gain
2
Computations based on changes in the dropout rates (1999-2000 and 1998-1999). Maximum # of dropouts in Yr. 1 & Yr. 2 divided by sum of of ABCs component scores. Difference (E) divided by standard deviation (G) yields the standard expected growth/gain (H).
3
If the composite is greater than or equal to zero, the school has made expected growth/gain.