Standards-based instruction, with the aim of grade-level achievement for all students, is undoubtedly the most comprehensive educational reform of the recent past. A hallmark of this reform effort is the measurement of student academic achievement with large-scale assessments that are used for accountability purposes. Assessment results are to be made public as a way of accounting for the academic achievement of all subgroups of students. Just as teachers, parents, and students are interested in individual student achievement, policymakers and the public in general are interested in student group achievement that indicates how specific schools, school districts, and states are performing. Never before have schools and states been under such scrutiny for demonstrating improved student outcomes for specific subgroups of students – students with disabilities, English language learners, students receiving free and reduced lunch, and students in general education.

Today’s widespread emphasis on statewide testing used for accountability purposes has essentially been driven by federal mandates. The Elementary and Secondary Education Act (ESEA) of 1994 included a strong mandate that all students, including those with disabilities, participate in states’ standards-based assessments and be counted in states’ accountability programs. Following a similar course in policy implementation, the 1997 amendments to the Individuals with Disabilities Education Act required the participation of students with disabilities in state and district-wide assessments. Most recently, the re-authorization of ESEA, No Child Left Behind Act (NCLB) of 2001, has re-focused states’ attention toward ensuring access to challenging, grade-level standards that are designed for the student’s grade of enrollment.

Reviewing the chronology of federal law that has strengthened the inclusion of students with disabilities in states’ large-scale assessment and accountability programs, and the new emphasis on grade-level standards, does not capture the political and controversial issues that have surrounded the implementation of assessments. This is certainly true for out-of-level testing, which refers to the practice of testing students below their grade of enrollment in states’ large-scale assessment programs. It is possible that no approach to testing has prompted such controversy at all levels of the American educational system – local, state, and federal – as has out-of-level testing.

Out-of-Level Testing Background

Including all subgroups of students in statewide testing has been challenging for states. In order to administer more inclusive large-scale assessments, 14 states in 2001-2002 added an approach to their testing program known as “out-of-level testing” so that some students could be tested at test levels below their grade of enrollment (Minnema & Thurlow, 2003). Policymakers, educators, and parents of students with disabilities thought that testing a student at the level on which teachers or others indicated that they were instructed in the classroom would yield more accurate, precise, and useful test results (Thurlow, Minnema, Bielinski, & Guven, 2003; Minnema & Thurlow, 2003). It was also thought that testing students on their “instructional-level” would be less frustrating and embarrassing since students could fully engage in completing test items. Other commonly held beliefs about out-of-level testing included that it would produce improved student motivation when taking tests, better attending behavior during test taking sessions, and enhanced student self esteem when students answered test items that tested content that they knew.

Also circulating in practice were attitudes and beliefs that discounted the value of out-of-level testing (Thurlow, Elliott, & Ysseldyke,1999). While referencing different reasons, other policymakers, educators, and parents thought that out-of-level testing would not yield more accurate, precise, and useful test results because students were tested on test material that was developed for much younger students (Thurlow & Minnema, 2001). Since the test material would most likely not be age appropriate, test motivation, attending behavior, and students’ self esteem could be adversely affected. Possibly the worst consequence of testing students with disabilities is the effects of setting lower expectations for students’ classroom performance or test level selection (Minnema, Thurlow, & Warren, 2004a). In addition, public reporting of out-of-level test results was particularly problematic because data managers were unclear as to how to report the test scores – on the grade of the student’s test level or grade of enrollment in school (Minnema & Thurlow, 2003).

The debate over the merit and worth of testing students with disabilities below their grade of enrollment continues to date. Researchers have begun to tease apart the complications of local and state level reporting (Minnema & Thurlow, 2003), uneven policy implementation (Minnema et al., 2004a), the prevalence of below-grade level testing (Thurlow et al., 2003), and other such issues that surround the implementation of out-of-level testing policies results. Nevertheless, research has yet to weigh in on the factual basis of many of the beliefs, attitudes, and perceptions that surface in educational practice.

In order to understand how states actually administered out-of-level testing policies at the local level, we designed a case study to look closely at local educational agencies where students with disabilities were tested below their grade of enrollment. We also sought to determine whether the many popular beliefs in practice about out-of-level testing were actually true. To meet these aims, we implemented two research studies in two school districts in two states. Both of these states were administering out-of-level tests as part of their large-scale assessment programs during the school year 2001-2002 when we collected our data.

This report is an accounting of a second case study of large-scale assessment practices in a local educational agency where students with disabilities were administered the state’s standards-based tests out of level. The first report (Minnema et al., 2004a) provided the results from the first case study conducted in another school district in another state. The overall purpose of our research project is to describe the specific effects of testing students with disabilities out of level as well as teachers’ and students’ perceptions of these effects.

State Context

In order to understand our qualitative and quantitative findings about out-of-level testing, it is important to first consider the statewide testing program that was used for below grade level testing in the state selected for the second case study. In 2001-2002, the large-scale assessment program included two criterion-referenced assessments designed to measure students’ mastery of the state’s academic content standards. One assessment measured academic achievement in the content areas of English language arts, mathematics, science, and social studies skills of students in 4^th and 8^th grades. This test was high stakes, meaning that students who scored below the approaching basic achievement level on both the English language arts and mathematics portions were not promoted to the next grade in school. A second test, also high stakes for high school graduation, assessed English language arts, mathematics, science, and social studies skills of students in 10^th grade. This test, a graduation equivalency exam, could be retaken if the student did not pass it. In order to receive a high school diploma, students were required to score at the approaching basic achievement level on both English language arts and mathematics sections of the assessment, as well as either the science or social studies portion. Additionally, a norm-referenced test was administered to students in the 3^rd, 5^th, 6^th, and 8^th grades, and another version of this norm-referenced instrument was administered to students in the 9^th grade. These tests were given as a statewide program of norm-referenced measurement of student academic achievement.

Students with disabilities in 3^rd through 9^th grade were eligible to participate in an out-of-level test in lieu of the criterion-referenced, standards-based measures administered in the 4^th and 8^th grades. The state’s out-of-level testing policy allowed for taking different content areas of the norm-referenced test at different grade levels according to the student’s abilities. Policy did require that either the English language arts or mathematics subtests be presented at least three grade levels below the student’s grade of enrollment. The science and social studies portions had to be administered at the same test level of either the English language arts or mathematics subtests. Administering all subtests on one grade level below the student’s grade of enrollment was also an option.

State policy required that students meet selection criteria for out-of-level testing and their Individualized Education Programs (IEPs) had to reflect the need for below-grade level testing. These criteria included scoring at the unsatisfactory level on the previous year’s general standards-based assessment in English language arts or mathematics or attaining a total score in reading, language, or mathematics at or below the fifth percentile on the norm-referenced test. In addition, a student’s IEP needed to reflect a functioning grade level in English language arts or mathematics that was at least three grade levels below the grade of enrollment in school. For these students, the norm-referenced test was administered below grade level instead of the general large-scale assessment.

A final condition that needed to be met in order to administer an out-of-level test was parental permission. Parents were required to sign a state-developed form that indicated that the family understood that by taking an out-of-level test, the child was not preparing academically for a standard high school diploma. In other words, a student who was not achieving on-grade level in elementary or middle school would not have the necessary academic skills to pass the graduation exit exam required for high school graduation.

 The School District

Data were collected in a school district in a southern state that serves approximately 18,595 students in grades pre-kindergarten through 12^th grade. The district contains 16 elementary, 6 middle, and 6 high schools. Students in this district are of predominately Caucasian (66.0%) and African-American (29.7%) ethnic backgrounds. Within the total student population, 41.4% of the district’s students were enrolled in a free and reduced lunch program. On average, the student to teacher ratio is 16.4 students to one teacher.

Four schools from this school district were considered for this case study. School A and School B were a middle school and a high school in one middle-sized city while School C and School D were a middle and high school respectively in a small neighboring community.

School A. The middle school that was located in the middle-sized city served approximately 691 students in 6^th through 8^th grades. Of the students attending this school, 33% received free or reduced-priced lunch. Class size varied, with 27% of classes having 1-20 students, 58% of classes having 21-26 students, and 15% of classes having more than 27 students. There were, on average, 16.9 students per teacher employed at the school. Students at School A came from a variety of ethnic backgrounds, including 65% Caucasian, 30% African-American, 3% Asian-American, 2% Hispanic, and <1% American Indian.

School B. The middle-sized city’s high school educated approximately 1,329 students in grades 9 through 12. In 46% of the school’s classes, class size consisted of 20 or fewer students, while 28% consisted of 21-26 students, and 26% consisted of 27 or more students. For this high school, the student-teacher ratio was approximately 16.8 students for one teacher on staff. Students receiving free or reduced-price lunch comprised 24% of the student enrollment. School B had an attendance rate of 91.1% and a dropout rate of 3.8%. The students were predominately Caucasian (71%), followed by African-American (25%), Asian-American (2%), Hispanic (2%), and American Indian (<1%).

School C. The second middle school in this case study was located in the smaller neighboring community, which served approximately 627 students in 6^th through 8^th grades. Of this school’s student population, 36% received free or reduced-priced lunch. Class size varied, with 16% of classes having 1-20 students, 48% of classes having 21-26 students, and 36% of classes having more than 27 students. On average, there were 17.4 students per teacher employed at the school. A variety of ethnicities were represented at School C, including 79% Caucasian, 20% African-American, 1% Hispanic, and <1% American Indian and Asian-American.

School D. There were approximately 631 students in 9^th through 12^th grades who attended this high school located in the neighboring, smaller-sized community. Class sizes consisted of 20 or fewer students in 48% of the classrooms with 31% of classes containing 21-26 students and 31% containing 27 or more students. There are approximately 17.1 students per teacher employed at the high school. Students who received free or reduced-priced lunch comprised 21% of students enrolled. The attendance rate for School D was 93.4% with a dropout rate of 4.9%. The student body was predominately Caucasian (81%), followed by African-American (18%), Asian-American (<1%), Hispanic (<1%), and American Indian (<1%).

Method

To study the implementation of an out-of-level testing policy at the local school level, we posed the following research questions:

(1)  What are the instructional effects on students with disabilities who are tested out of level in statewide assessments?

(2)  What are teachers’ learning expectations for students with disabilities who are tested out of level?

(3)  How are students with disabilities selected for an out-of-level test?

(4)  How do students and their parents perceive out of level testing?

Research design. We used a mixed methods approach to implementing our case study where a “case” was defined as a school district. Although the school district was relatively large, limited resources constrained our data collection process to a limited number of participating schools. Two district-level personnel, a special education director, and a special education coordinator, selected and recruited two middle schools and two high schools to participate in our study. No elementary schools participated because this state did not administer out-of-level state tests to elementary-aged students. At the school and district levels, we collected both narrative and numeric data through interview, survey, and document review data collection techniques.

Sample. The special education coordinator invited all special and general education teachers and administrators from each school to participate in our case study. Our purposive sample consisted of special education teachers (n = 15), general education teachers (n = 10), and principals (n = 3). In terms of a district level perspective, we also included a district level test coordinator in our data collection activities. Each participant received a gift certificate from a local department store as a thank you for the time invested in our case study.

Instruments. Our data collection techniques included face-to-face interview protocols and a document review data collection sheet. The face-to-face interviews required approximately 25 minutes for school personnel. The purpose of these educator interviews was to garner their opinions about and their perceptions of student experiences in out-of-level testing (see Appendix A for copies of the interview protocols). The purpose of the IEP document reviews was to describe instructional features of those tested out of level. All instruments were pilot tested prior to collecting data.

Procedures. Two NCEO researchers collected all data on-site in the schools. The student interview protocol was reviewed with a NCEO researcher prior to conducting the student interviews to ensure an appropriate interview process. To introduce the research project to district and school staff, we recruited one contact person at the district level. She agreed to read a letter from NCEO explaining the research project prior to our visit. Once on site, she helped distribute and collect written surveys as well as access student IEPs for the document review. In addition, the contact person worked with school staff to schedule the face-to-face interviews.

A data collection activity was begun during the study to address the instructional effects of testing students with disabilities below their grade of enrollment because of contextual factors beyond our control. After data collection had been initiated, we learned that the school district was under a decree from the U.S. Office for Civil Rights to provide instruction to all students at the same pace in every general educational classroom. In other words, all students, including students with disabilities who were tested out of level, progressed in curriculum in all content areas at the same pace whether or not the necessary information had been learned. Rather than collecting interview data to answer our research question about the instructional effects of out-of-level testing, we created “scenarios” that described typical students in their classrooms based on conversations with general and special education teachers. We also asked teachers to describe special education instruction as it was delivered in their school. These discussions became part of our interviews, so were tape recorded and transcribed. These data were analyzed for information about the instruction and learning of students with disabilities who were tested out of level.

Data analysis. We used two basic approaches to analyze the case study data. For qualitative data analysis, all educator interviews were tape recorded, transcribed, and subjected to a content analysis that yielded thematic findings. Since the student interview responses were briefer than the educator responses, we tabulated by response content within the student data set to create frequencies of types of student responses. IEP review data were analyzed with descriptive statistics.

Findings

Our findings are organized according to the three data collection activities employed for this case study. The IEP document review is presented first with the scenario results presented second. Face-to-face interview results are presented third.

IEP Document Review Results

We reviewed students’ IEPs from Schools A, B, and C where students were tested out of level in statewide testing over three school years. Since no students were tested out of level in School D, the smaller-sized city high school, we did not review IEPs there.

A summary of the number of out-of-level tests in each school is shown in Table 1. In School A, a total of 21 out-of-level tests were administered with 9 tests given in school year 1999-2000, 8 tests given in school year 2000-2001, and 4 tests given in school year 2001-2002. Three students with disabilities were tested out of level all three years with one of those students tested only in math two of the three years. Other students were tested in both reading and math for either one or two years. Four out-of-level tests were taken in School B during school year 1999-2000, one out of-level test during school year 2000-2001, and no out-of-level tests in 2001-2002. There were four out-of-level tests given in School C during school year 1999-2000, one out-of-level test during school year 2000-2001, and no out-of-level tests during school year 2001-2002. No students in either school B or school C were tested out of level all three years.

Table 1. Number of Out-of-Level Tests by School Year by School

 * Number of students with disabilities tested out of level for all three years.

For each school, we determined the number of grade levels below the students’ grade levels of enrollment that out-of-level tests were administered in reading/language arts and math. These results are presented in Table 2 for each school. Overall, the numbers of out-of-level tests for the two content areas were about the same.

As shown in the table for reading/language arts, most of the tests taken out of level in School A were 4 grades below the grade of enrollment (n = 8). Others ranged between 3 and 6 grade levels below. Of the 3 tests administered out of level in School B, test levels were 4, 5, and 6 levels below students’ grade level in school. There were also 4 out-of-level tests in School C with 1 test at each level from 2 to 5 levels below the grade of enrollment.

Of the 15 math tests administered out of level in School A, the largest number of tests (n = 5) were given 4 levels below grade level. The remaining tests were administered from 1 level below to 6 levels below grade level. In School B, 1 math test was administered 4 levels below and 1 test at 5 levels below; 2 tests were administered 7 grade levels below. The number and test levels of math tests given in School C were spread from 1 level below to 5 levels below grade level.

Table 2. Number of Test Grade Levels Below Students’ Grade of Enrollment by Test and School

Student descriptors were gleaned from students’ IEPs for each school. We developed summaries of students’ disability categories, grades of enrollment, the grade level at which students tested in reading/language arts and math, and the “instructional grade level” for reading/language arts and math. The instructional grade level was determined by the results of norm-referenced tests that were administered to determine the test grade level at which an out-of-level test should be given. Most of the students received the Kaufman Test of Educational Achievement, 2^nd Edition (KTEA-II), although a few students received either the Iowa Test of Basic Skills (ITBS) or the Woodcock-Johnson-Revised (WJ-R) to determine an out-of-level test level.

Table 3 shows the student descriptors for the middle school located in the middle-sized city, which had 11 students with disabilities tested out of level across the school years that we studied. Eight of the students tested out of level had learning disabilities (LD) while three students had mild cognitive disabilities (MCD). Two of the students with LD had secondary disabilities, one being a speech/language disability (SPL) and the other a hearing disability (HD). Students in School A were tested out of level for one, two, or three school years. Specifically, four students were tested out of level for only one school year; these students included one student in 7^th grade and three students in 8^th grade. Six students were tested out of level in two consecutive school years. One student was tested in both 5^th and 6^th grade, another student in 6^th and 7^th grades, and three students tested in 7^th and 8^th grades. One student repeated 7^th grade and was tested out of level in both of those school years. Three students were tested out of level over three consecutive school years in 5^th, 6^th, and 7^th grades.

In reading/language arts most of students in School A were tested at or close to their instructional level as determined by a standardized test. In fact, two students were tested on the grade of enrollment in reading/language arts and four students were tested on the grade of enrollment in math. The exception was student 1 who was tested at grade level 5 in the first year and grade level 6 in the second and third years when the standardized test indicated an instructional level of 1.9 in the first year of testing. One other student was tested one grade level above the grade level indicated by a standardized test.

Compared to reading/language arts testing, more students were tested more grade levels above the instructional level in math testing. Four students were tested either 3 or 4 grade levels above the level indicated by the standardized test. These students were in 6^th, 7^th, or 8^th grade in school, but scored at approximately the 2^nd or 3^rd grade levels in math. All of these students were tested out of level in math at test levels closer to their grade of enrollment in school. There was one student, student 3, who was tested at the 7^th grade level when in 6^th grade and the 6^th grade level in the 7^th grade. Reading/language arts tests were administered in the same way with the second year of testing dropping down a test level. Standardized testing during the second year indicated that the out-of-level test could be a grade level lower. While seeming to be inaccurate, both researchers recorded these instructional and testing levels according to the student’s IEP.

Table 3. Out-of-Level Tested Student Features by Student in School A