Since the advent of standards-based reform with federally-mandated statewide testing as a means to measure student and system achievement toward grade-level academic standards, states have struggled with the best ways to include students with disabilities. Most students with disabilities take the regular assessment with or without accommodations, and about 1% of the total population of students participate in an alternate assessment developed for students with significant cognitive disabilities (approximately 10% of the population of students with disabilities). Yet, states have still expressed concerns about the appropriateness of the assessments (Almond, Quenemoen, Olsen, & Thurlow, 2000). To better include these students in large-scale assessments, states have added other testing options to their statewide testing program.

One alternative option to the regular assessment with or without accommodations or the alternate assessment that is used in some states (14 in 2000-2001) is testing students with disabilities using tests intended for students at a lower grade level. This option is often called out-of-level testing. Out-of-level testing is a controversial and politicized approach to standards-based assessment (Thurlow & Minnema, 2001). Historically, out-of-level testing was thought to reduce student test anxiety, yield a more accurate measure of student academic achievement, and allow more students with disabilities to be included in state testing and accountability programs. However, recent work at the National Center on Educational Outcomes (NCEO) has demonstrated that out-of-level test scores are rarely reported publicly (Minnema & Thurlow, 2003) or used for either student or system accountability purposes. There are also no data that definitively determine whether student test anxiety is actually reduced during below-grade level testing. In fact, in case studies (e.g., Minnema, 2003), teachers reported that students with disabilities are embarrassed when taking a test that is lower than the test level of their peers. In addition, research has not sorted out the accuracy and precision of out-of-level test results in terms of measuring students’ progress toward academic standards.

Besides the lack of definitive research, clarification of the issues that surround out-of-level testing is also complicated by the variety of testing options that states have developed. In fact, there are several closely related approaches to non-grade-level testing that may or may not be viewed as the same thing as out-of-level testing (e.g., levels testing), making it difficult for the field to sort out what really is out-of-level testing and what is not. To complicate the situation further, it is difficult to find any non-grade-level test results in states’ data reports. In other words, there is a lack of information about the number of students who are involved in any type of non-grade level testing within states that have standards-based assessments (Minnema & Thurlow, 2003). Further, states’ data reports do not include information that describes at what grade levels students with disabilities were tested by non-grade level tests. Without these test data, patterns in students’ test performance cannot be ascertained that point to the appropriateness of the test levels administered. We view these data – the number of students tested on non-grade-level tests (prevalence) and the performance of these students – as necessary first steps toward understanding how states are including students with disabilities by administering non-grade-level tests.

The purpose of this study was to analyze data from states that offer out-of-level testing for students with disabilities. Analyses were conducted to determine, first, the prevalence of students with disabilities participating in non-grade-level testing options. Specifically, we examined test results from three states for the school year 2000-2001. As a second step in data analysis, we examined the overall performance patterns in the test results to begin to evaluate the appropriateness of administering non-grade-level tests to specific groups of students with disabilities. The study had two research questions:

Each of the 14 states that used out-of-level testing in its statewide testing program during the 2000-2001 school year was invited to participate in this study. States could either send raw data files for NCEO researchers to analyze, or they could provide state-analyzed test results for NCEO to use. For states that had not yet determined how to make out-of-level test scores public, we requested special data runs of their out-of-level test results.

Three states consented to provide data. The other states either were not interested in participating or had a variety of issues related to identifying out-of-level test results on a statewide basis. Two of the states that provided data had analyzed data on their own and publicly reported those data. For example, one state analyzed its data and distributed a report statewide for districts to review and examine local patterns of out-of-level test results. The publicly distributed data were examined and included here. Another state analyzed data specifically for NCEO in response to its request.

We varied our analyses of data for prevalence information based on the nature of each state’s data. Generally we used frequency counts, and percentages if possible, to describe the prevalence of testing below grade level for each content area tested in a state. In some states, however, information was available only for the grade in which the test was administered, and not for the grade in which students were enrolled. In some states, it was the opposite – data were available for the students’ grades of enrollment, but did not indicate the specific grade level of the test that was taken. The nature of the prevalence data is clarified in the presentation of each state’s data.

To examine test difficulty for those students tested out of level, we examined states’ performance data as a reasonable indicator of whether a student was appropriately challenged. The criteria that we used to indicate that a test was "too hard" or "too easy" were based on the available performance data. Because states’ data varied widely across the three participating states, we used two different approaches to deduce the overall difficulty of out-of-level tests. In State One, we used an indication of the proficiency levels attained by students. However, we did not have this type of indicator to judge test difficulty for out-of-level tests administered in State Two and State Three. In the other two states, we examined the percentage correct obtained by the students. We selected criteria of fewer than 30% correct as a proxy for a test that was "too hard" and more than 80% for a test that was "too easy." For these, a student was considered to have scored "quite low" if the student answered <30% of the items correctly. A student was considered to have scored "quite high" if the student answered >80% of the items correctly.

The results of our analyses are presented below for each state separately. States’ names are not reported here, but a general description of the assessment system in each state as it existed in 2000-2001 is provided to give context to the results.

In this state, special education students who were thought to be unable to participate in the standard statewide assessment program had the option of participating in one of two alternate assessments. Alternate Assessment Option 1, out-of-level testing, was designed for students who had not received any grade-level instruction on skills covered on the regular state assessments. These students typically had moderate disabilities and had been instructed below grade level over consecutive school years; a standard test administration at these students’ assigned grade levels was thought to result in invalid assessments of their academic achievement. Therefore, the test was administered two or more grade levels below the grade in which the student was enrolled (e.g., tenth-grade student takes grade 8 test). This option was not available for fourth-grade students for the writing test. The second option was Alternate Assessment Option 2, which was a skills checklist designed for students who do not participate in an academic curriculum due to severe disabilities.

The decision of which assessment option to use with each student was made by the student’s Individualized Education Program (IEP) team and was reiterated in the student’s IEP. It was expected that about 15 percent of special education students would participate in Alternate Assessment Option 1 (out-of-level testing) and approximately 5 percent of special education students would participate in the Alternate Assessment Option 2 (skills checklist).

Prevalence of Out-of-Level Testing. Table 1 displays the number of students in special education enrolled in each grade who were tested out of level in reading, math, and writing (i.e., 28% of grade 8 special education students taking the reading test were tested out of level) in 2000-2001 in State One. Overall, approximately 30% of the special education students were tested out of level in reading and math across test grades 4, 6, and 8. Fewer students, approximately 20%, were tested out of level in writing. This was probably due to the lack of a writing prompt at Grade 2.

Presented in Table 2 is the number of out-of-level tests at each grade and the percentage of students by grade at which they were tested. For instance, of the 1,794 6th grade students who were tested out of level, 32% tested down two levels below their assigned grade level meaning that they took the 2nd grade test. Overall, the largest number of students tested out of level was in the 6th grade. More specifically, for each grade, the largest number of students was tested two grade levels below their grade of enrollment.

Placement Accuracy. In Table 3, we show the percentage of students at each grade level scoring at or above the goal. Scoring at this level would indicate that the test probably was too easy for them. As is evident in the table, approximately 30% at each grade level (4, 6, 8) who took the 2nd grade test scored at or above goal (level 3). The percentage dropped to about 10% for the grade 4 and grade 6 tests.

* 1 = Intervention Level

** 3 = Goal Level for Grade 2 Test

*** 4 = Good Level for Grades 4 and 6 Tests

Similar data were presented by State One for the content areas of mathematics and writing. These data are summarized in Table 4, which lists just the percentages of students in each grade of enrollment who performed at the intervention level and the goal level. As is evident in this table, in each grade there were some students who performed at or above goal level. It is also evident, however, that there were increasing percentages of students at each grade level who performed at the intervention level.

For students "challenged" against their enrolled grade level benchmark, students were assigned to the level best aligned to their ability as indicated by the four criteria: (a) student’s performance from a prior grade, (b) 20-item locator test, (c) results from a sample test provided by the state, and (d) professional judgment.

Prevalence of Out-of-Level Testing. Prevalence and performance results for State Two’s Reading Literature test are displayed in Table 5. The table shows the numbers of students with disabilities taking each grade-level test that was a benchmark below their enrolled grades. Overall, of all students tested on Benchmarks 1, 2, and 3, 12% (n=1,344) were actually enrolled in a grade level above the benchmark on which they were tested. The percentage of students taking each benchmark who were enrolled in higher grades decreased as the benchmark increased. Thus, 19% (n=730) of all students taking Benchmark 1 (n=3,794) were actually enrolled in higher grade levels; for those taking Benchmark 2 (grade 5), the percentage decreased to 11%, and by Benchmark 3 (grade 8), the percentage was 4%.