Standards-based instruction, with the aim of grade-level achievement for all students, is undoubtedly the most comprehensive educational reform of the recent past. A hallmark of this reform effort is the measurement of student academic achievement with large-scale assessments that are used for accountability purposes. Assessment results are to be made public as a way of accounting for the academic achievement of all subgroups of students. Just as teachers, parents, and students are interested in individual student achievement, policymakers and the public in general are interested in student group achievement that indicates how specific schools, school districts, and states are performing. Never before have schools and states been under such scrutiny for demonstrating improved student outcomes for specific subgroups of students – students with disabilities, English language learners, students receiving free and reduced lunch, and students in general education.

Today’s emphasis on statewide testing that is used for accountability purposes has essentially been driven by federal mandates. The Elementary and Secondary Education Act (ESEA) of 1994 is a strong mandate that requires that all students with disabilities participate in states’ standards-based assessment and be counted in states’ accountability programs. Following a similar course in policy implementation, the Amendments to the Individuals with Disabilities Education Act (1997) first emphasized the inclusion of students with disabilities in large-scale assessment programs. Most recently, the re-authorization of ESEA, No Child Left Behind Act (NCLB) of 2001 has re-focused states’ attention toward ensuring access to challenging, grade-level standards that are designed for students’ grade of enrollment.

NCLB (2001) is currently the most stringent in requiring that all students be measured to grade-level criteria so that every subgroup of students receives challenging, standards-based instruction based on the grade in which these students are enrolled in school. Nevertheless, reviewing the chronology of federal law that has strengthened the inclusion of students with disabilities in states’ large-scale assessment and accountability programs does not capture the political and controversial issues that have surrounded the implementation of federal mandates. This is certainly true for out-of-level testing, or the practice of testing students with disabilities below their grade of enrollment in states’ large-scale assessment programs. Possibly no approach to testing has prompted such controversy at all levels of the American educational system—local, state, and federal—than out-of-level testing.

Out-of-Level Testing Background

Including all subgroups of students in statewide testing has been challenging for states. In order to administer more inclusive large-scale assessments, 14 states in 2001-2002 have added an approach to their testing program known as “out-of-level testing” so that some students could be tested at test levels below their grade of enrollment (Minnema & Thurlow, 2003). Many arguments have been used to justify out-of-level testing. Policymakers, educators, and parents of students with disabilities thought that testing a student at the level on which they were instructed in the classroom would yield more accurate, precise, and useful test results (Thurlow, Minnema, Bielinski, & Guven, 2003; Minnema & Thurlow, 2003). It was also thought that testing students on their “instructional-level” would be less frustrating and embarrassing since students could fully engage in completing test items. Other commonly held beliefs about out-of-level testing included improved student motivation when taking tests, better attending behavior during test taking sessions, and enhanced student self esteem when students answered test items that tested content that they knew.

Also circulating in practice were attitudes and beliefs that discounted the value of out-of-level testing. While referencing different reasons, other policymakers, educators, and parents thought that out-of-level testing would not yield more accurate, precise, and useful test results because students were tested on test material that was developed for much younger students. Since the test material would most likely not be age appropriate, test motivation, attending behavior, and students’ self esteem could be adversely affected. Possibly the worst consequence of testing students with disabilities is the effects of setting lower expectations for students’ classroom performance or test level selection. In addition, public reporting of out-of-level test results was particularly problematic because data managers were unclear as to how to report the test scores—on the grade of the student’s test level or grade of enrollment in school.

The debate over the merit and worth of testing students with disabilities below their grade of enrollment continues to date. Researchers have begun to tease apart the complications of local and state level reporting, uneven policy implementation, the prevalence of below-grade level testing, and other such issues that surround the implementation of out-of-level testing policies results (Thurlow, et al., 2003; Minnema & Thurlow, 2003). Nevertheless, research has yet to weigh in on the factual basis of many of the beliefs, attitudes, and perceptions that surface in educational practice.

In order to understand how states actually administered out-of-level testing policies at the local level, we designed a case study to look closely at local educational agencies (LEAs) where students with disabilities were tested below their grade of enrollment. We also sought to determine whether the many popular beliefs in practice about out-of-level testing were actually true. To meet these aims, we implemented two research studies in two different school districts in two different states. Both of these states were administering out-of-level tests as part of their large-scale assessment programs during the school year 2001-2002 when we collected our data.

This report is the first accounting of one case study of large-scale assessment practices in a local educational agency (LEA) where students with disabilities are administered states’ standards-based tests out of level. A second report (Minnema, Thurlow, & Warren, 2004b) provides the write up of the results from the second case study conducted in another school district in another state. The overall purpose of our research project is to describe the specific effects of testing students with disabilities out of level as well as teachers’ and students’ perceptions of these effects.

State Context

In 2001-2002, the large-scale assessment program for the state chosen for the first case study was an augmented version of the Stanford Achievement Test, Ninth Edition (SAT-9) where items that directly measured state content standards were added to this norm-referenced test. More specifically, these additional test items were designed to measure students’ progress on acquiring content standards in English-language arts, mathematics, science, and history/social science in grades 2 through 11. An augmented SAT-9 was included in the English-language arts and mathematics portions of the state test by selecting certain SAT-9 items that closely aligned with the state’s content standards. The complete battery of the national norm-referenced SAT-9 was also given to students in grade 2 through 11, assessing reading, language (written expression), and mathematics. Students in grades 2 through 8 were assessed in spelling, and students in grades 9 through 11 were assessed in science and social science using the SAT-9. Also, state writing tests were administered in grades 4 and 7. In addition to the English version tests, the standards-based assessment and the SAT-9 tests, a Spanish Assessment of Basic Education, Second Edition (SABE/2) was used to assess Spanish-speaking students in reading, spelling, language, and mathematics in grades 2 through 11. These students must have been identified as a limited-English proficient student who had been in school for less than 12 months.

This state offered an out-of-level testing option for students with disabilities whose Individualized Education Program (IEP) documented a need for below grade level assessment. These students participated in the statewide testing program by taking the state standards-based test and the SAT-9 at any available level below the student’s grade of enrollment. One level below the student’s grade of enrollment was considered a standard test administration, while two or more levels below the student’s grade of enrollment were considered a non-standard administration.

The School District

Data for this case study were collected in a unified school district located in the northern region of a large western state. The district served approximately 16,881 kindergarten through grade 12 students in 21 elementary, four middle, and four high schools. The mission of the district is to “produce educated citizens who achieve and perform at all levels of learning, are prepared to live fulfilling lives, and contribute to their community and the world in which they live.” The student ethnicity of the district includes 53% Caucasian, 38% Hispanic, 3% Filipino, 2% African-American, 2% Asian-American, and 2% American Indian.

Within this large school district, two middle schools (Schools 1 and 2) and one elementary school (School 3) were studied. School 1 served approximately 654 students in grades 6 through 8 who lived in a neighboring small city to that of School 2. Average class size was 27 students per class. School 3 was the most culturally diverse of the three schools studied, with a student population of 48% Caucasian, 19% Hispanic, 12% Filipino, 11% African-American, 3% Asian-American, 3% Pacific Islander, and 1% American Indian/Alaskan Native. Of the total student body, 8.7% of the students in School 3 were classified as limited English proficient. School 1 had four special education classrooms, two special day classes for students with cognitive disabilities and two classrooms for students with learning disabilities. Students from each of these classes were integrated into general education classes with levels of special education support as necessary.

School 2 was located in the city proper, and served approximately 1,013 students in grades 6 through 8. The average class was 28 students. There were more special education programs housed in School 2 than in the other two schools, two of which are a therapeutic day class for students with emotional and behavioral disabilities and a segregated classroom for students who have cognitive disabilities.

School 3, the elementary school, was located in a rural area of a relatively populated area of this state. Approximately 207 students attended this school in grades K-6. In School 3, the primary instructional focus was literacy for approximately 207 students in grades K-6. The curriculum was guided by this state’s content standards so that every child has the opportunity to learn grade level standards. Average class sizes for grades K-3 was 18 students while the average class size for grades 4-6 was 26 students.

Method

Research Questions

Our research project addressed the following research questions:

(1) What are the instructional effects on students with disabilities who are tested out of level in statewide assessments?
(2) What are teachers’ learning expectations for students with disabilities who are tested out of level?
(3) How are students with disabilities selected for an out-of-level test?
(4) How do students with disabilities perceive out of level testing?

Sample

Our purposive sample included students with disabilities (n = 14), general education teachers (n = 5), special education teachers (n = 8), school administrators (n = 3), special education coordinators (n = 1), and other school staff such as guidance counselors (n = 1) and district test coordinators (n = 1). These participants were employed by a school district that was recommended by the state educational agency. The schools were self-selected by the district test coordinator and director of special education. Each participating school agreed to participate in the case study. All participants received a gift card for a local department store with the amount dependent upon the amount of time invested in our research activities.

Research Design

We used a case study design to address our research questions by employing mixed methods to garner numeric and narrative data.

Instruments

Our data collection techniques included face-to-face interviews (n = 33) and a document review of students’ Individualized Education Programs (n = 14) and students’ school records (n = 54). The data collection activity, face-to-face interviews, required approximately 25 minutes for school personnel to complete and less time for students with disabilities to complete. The purpose of these educator interviews was to garner their opinions about and their perceptions of student experiences in out-of-level testing while the purpose of the student interviews was to learn directly from them how they perceived their test experiences (see Appendix A for copies of the interview protocols).
 

Procedures

We designed our case study to collect interview and document review data on-site in each participating school and school district offices. One person from each school served as a contact person to assist in scheduling interview appointments and distributing the written surveys.

Data Analysis

We used two basic approaches to analyzing our case study data. For our qualitative data analysis, all educator interviews were tape recorded, transcribed, and subjected to a content analysis that yielded themes of results. Since the student interview responses were briefer than the educator interviews, these interviews were not tape recorded. Instead, student responses were written down during the interviews. To analyze the student interview data, we tabulated categories of responses rather than creating themes of results. In terms of our numeric data, we used descriptive statistics to analyze the IEP review data.

Findings

IEP Document Review

IEPs were reviewed in one middle school (School 1) where 25 students were tested out of level. Of these 25 students, 14 parents granted permission for us to review their child’s IEP. Special education teachers provided some information for the remaining 11 students. In a second middle school (School 2), 54 students with disabilities were tested out of level. A special education coordinator provided some data for these students. In the remaining one school, only two students were tested out of level. These students were not included in the data collection activity because the small number does not support an analysis at the school level.

Table 1 shows the numbers of students tested out of level as a function of their grade, disability, and subject. Most students tested out of level in School 1 were assigned to the 6th grade; however, grade level was not provided for approximately half of the students. Within School 2 where we had grade assignments for all students, the majority of students tested out of level attended 7th grade.

In terms of disability category, most students tested out of level in both schools had learning disabilities. This finding may be due to the national trend in which students with learning disabilities is the largest category of students identified for special education services (U.S. Department of Education, 2001).

Regarding special education setting, more students attending Resource Classes in School 1 were tested out of level while more students placed in Special Day Classes were tested out of level in School 2. Some of the 7th grade students in School 2 attended core content classes in both resource and special day classes. Since these students are neither resource nor special day class students only, they are presented as “combined” in the table below. Combined placements such as these did not occur for the 8th grade students in School 2 or for any students in School 1. Several students from School 2 were placed in specialized therapeutic programs or alternative middle school classes. The number of students tested out of level from these programs is designated as “Other” in Table 1.

Table 2 demonstrates that the majority of out-of-level tests administered to middle school students were at least three grade levels below the grade in which the students were enrolled in school. In School 1, 19 of 25 tests (with 2 students missing a test level) were presented at the 2nd grade through 5th grade level. Only four students were tested at either 6th or 7th grade. While School 2 had 15 of 54 students with disabilities tested at the 6th grade level, 37 of 54 tested received state tests at either the 2nd, 4th, or 5th grade level.

Table 1. Number of Students Tested Out of Level by Grade, Disability, and Setting
 

Table 2. Grade Levels Administered as Out-of-Level Tests

The data in Table 3 show differences between School 1 and School 2 in terms of the number of levels below grade level tested by out-of-level tests, and whether entire tests (or partial tests) were administered out of level. In School 1, all of the out-of-level tests were administered as entire tests. An equal number (n = 11) were presented 1 to 2 levels below grade level as were presented 3 to 5 levels below the students’ assigned grade levels. One test was administered 5 levels below grade level and no tests were administered 6 grade levels below. There were three students in School 1 with missing data. In School 2, 31 of 54 out-of-level tests were tested close to the students’ assigned grade level (i.e., 1 or 2 levels below). Of the 29 partial out-of-level tests, 16 were administered either 3 or 4 levels below grade level while 13 tests were administered either 5 or 6 levels below.

Table 3. Number Levels Tested Below Grade Level by Entire or Partial Test

Even though the number of IEPs reviewed (n = 14) is relatively small, the results remain interesting in that there is high variability between reading and math instructional grade levels when compared to the grade levels at which the students were tested. In this school, teachers indicated during the face-to-face interviews that out-of-level test levels were set according to students’ academic strengths as demonstrated by academic progress in classroom performance. When comparing teacher determined instructional levels in students’ reading and math skills to the grade levels at which they were tested in reading and math, most of the students (n = 13) were not tested at appropriate levels according to the grade level that teachers identified as their academic strength. For instance, one 6th grade student, whose teacher identified as reading at a 2nd grade level, was tested at the 3rd grade level even though math abilities were identified to be a the 5th grade level. When using the criterion of testing a student at his or her teacher-identified grade levels in reading and math, only one student’s test levels matched his or her instructional grade level because the teacher-identified reading and math levels were set at the same grade level.


Table 4. Grade, Content Area, and Test by Levels for School 1

Face-to-face Interviews

Student Interviews

We interviewed middle-school students with disabilities (n = 10) who attended special day classes or resource classes. Some students were included in general education instruction with paraprofessional support. Interview results included the following:

• Most students said that they liked taking the statewide test out of level (8 out of 10 students). Half of the students thought it was neither too hard nor too easy, although two students described the below grade-level test as “too babyish.” When asked about test rigor, seven students mentioned guessing at item responses, although six of these students indicated that they guessed only minimally and one student guessed frequently. Only one student indicated guessing at all throughout the out-of-level test. Two students did not mention guessing at test item responses.

• Only four students were able to appropriately describe an out-of-level test as being a test at a grade level below that which they were enrolled in school.

• Only one student knew that someone on the IEP team made the decision to test out of level. But even though this student attended the IEP team meeting, the student did know if a parent or teacher made the decision.

• None of the students’ responses indicated that they understood how out-of-level testing could affect their future school experiences. It is interesting to note that six of the ten students plan to graduate from high school with three planning to receive a regular high school diploma. Of these six students, three planned to attend college. Of the remaining four students interviewed, two of them have set post-high school occupational goals.

Special Education Teacher and Administrator Interviews

The results of our face-to-face interviews with teachers and administrators are presented by themes of results. The thematic results are divided into three topical areas: (1) comparing out-of-level testing to on-grade level testing, (2) selecting students and test levels for out-of-level tests, and (3) interesting aspects of policy implementation. Included in the teacher interviews were the three special education coordinators who also had caseloads of students with disabilities for whom they provided services.
 

Comparing Out-of-level Testing to On-grade Level Testing

In Table 5 we present the results from the interview questions focused on the benefits and concerns of out-of-level testing. These narrative data revealed that there are varied opinions about out-of-level testing that do not fall into an orderly pattern. Both teachers and other school staff (e.g., principals, guidance counselor) identified benefits and concerns about testing students with disabilities out of level. In one case, the same idea, “negative impact on students’ self-esteem,” was identified as a concern for both out-of-level testing and on-grade level testing. Of importance in the teachers’ responses is the concern that an out-of-level test does not document achievement toward grade-level standards. Teachers suggested further that this is particularly true if students are continually tested out-of-level at the same grade level. Teachers also highlighted the concern that “sometimes they [students with disabilities] are tested in one area that maybe is below their ability, but that’s the way the tests are given. They’re all given at one grade level.” Another concern that was reflected in teachers’ responses was the lack of usable test results because only raw scores are provided for out-of-level tests given at more than one level below students’ grades of enrollment. These scores do not provide the normative information necessary to make instructional decisions. It is interesting to note that administrators identified concerns about out-of-level testing even though this question was not posed to them. These concerns parallel the concerns raised by the teachers.

Table 5. Comparing Out-of-level and On-grade Level Testing:  Benefits and Concerns

The responses to the interview questions that compared student behavior during out-of-level testing to on-grade level testing fell into a clear pattern of results when the test environment was considered (see Table 6). Teachers identified inappropriate test behaviors during both out-of-level testing and on-grade level testing. In contrast, teachers identified appropriate test behavior during out-of-level testing, but not during on-grade level testing. Inappropriate test behaviors were said to occur during out-of-level testing only when multiple levels of the same test were presented within the same classroom. In other words, when students could compare their test level to the test level of other students, their behavior tended to be disruptive. Some teachers noted that during this testing situation, some students appeared to feel badly about having a test level lower than the other students who were testing in that classroom. While this interview question was not part of the administrators’ interview protocol, some administrators commented that they were not aware of student behavior during out-of-level or on-grade level testing.

Table 6. Comparing Out-of-level Testing to On-grade level Testing: Student Behavior

Selecting Students and Test Levels for Out-of-Level Tests

One of our research questions pertained to the selection of students with disabilities for out-of-level testing. In order to answer this research question accurately, it is important to first consider the educational context in which these assessment decisions were made. Most students in School 1 and School 2 who attended resource classes had learning disabilities and were participating in mainstream education with paraprofessional support during instruction. Generally speaking, students who received special education services in special day classes had more severe disabilities so that little to no instruction occurred in general education classrooms. Within this context, it seemed that an underlying assumption was driving the decision to test students with disabilities out of level in both schools. Teachers who taught special day classes generally believed what was reflected in the following statement made by a special day classroom teacher: “Most of our students are below grade level, so we know that a grade level test would be really hard for them to do, or next to impossible. Most of them are three to four years behind grade level. To give them one grade level lower doesn’t really help that much.” Another participant indicated that, “even if they’re in a regular class, sometimes the level of work they’re getting is not 7th grade or 6th grade level, but more like 4th or 5th grade.”

In terms of selecting students for out-of-level testing, each educator interviewed indicated that the decision to test a student out of level was discussed and decided during the student’s IEP team meeting. It generally occurred near the end of the meeting when the IEP paperwork was completed. The team case manager writes in the IEP the decision to test out of level that requires a parent initial indicating agreement. Participants indicated varying levels of active discussion in making the decision to test below grade level. For instance, one participant commented that “the IEP team determines that, but honestly, it would boil down to a lot of input from the special education team. A lot of it has been my decision.” On the other end of the continuum, an administrator suggested that “it happens at the students’ IEP, where the parents are involved. If the student is so delayed, where he’s working two grade levels behind, then the topic is really raised.”

According to the themes from our interviews, four factors were considered in determining whether a student should be tested on grade level or out of level. First, teachers thought about students’ “ability levels” based on “what their [academic] strengths are.” Typically, this was based on their “functioning levels in the classroom and their work samples.” Second, “educational assessment results for individual students” are considered that point to specific grade levels of ability. Third, parent considerations are also part of the decision-making process, meaning that “the parents actually make the decision after we [special education teachers] counsel them on … what grade level the kid is reading at and what we think he would do well on.” “Sometimes parents will want this kid tested at grade level … because they think it will help motivate the kid.” But, “most of the time the parent goes with what we suggest.” One participant suggested that in making the decision to test out of level, “most parents are worried … so they go by how frustrated their child is.” Finally, “if you go just one grade level down, the test still counts as a standard presentation. Sometimes that factors in.”

There is also a generally accepted process among both teachers and administrators to select a grade level at which to administer an out-of-level test. When thinking about students’ level of academic functioning in determining the need to test below grade level, the grade at which to test is also considered. Teachers do this in two ways. First, “it’s based on their ability level,” which is determined “by the assessments, either standardized or non-standardized, that I do and by the discussion with the teachers about their functioning level in the classroom.” A test level is then selected according to “… what grade level we feel they could take the test and still be a little challenged, but also be able to succeed in it.”

Students new to the school district tended to be exceptions to the IEP decision-making process. In one case, a special education teacher reported calling a parent of a new student on the telephone to say, “The [statewide tests] are coming up. Your child just tested at this grade level, and I think it would be a good idea for us to let her keep taking tests at that same level. The parent said, ‘Oh. OK.’” This student was the only student in resource classes whose selection for out-of-level testing was reported as based on a teacher recommendation with parental passive acceptance. Teachers and administrators also indicated that students new to this school district entered with the decision to test out of level already made by the previous IEP team.
 

Aspects of Policy Implementation at the School Level

There are also interesting aspects to the manner in which this assessment policy is implemented in the schools. For both schools, two patterns emerged in the teachers’ and administrators’ responses that highlight differences in testing practice according to students’ educational placements and students’ grade levels. These considerations are presented in Tables 7 and 8.

Table 7. Differences by Student Placement

Table 8. Differences by Student Grade of Enrollment

Regarding students’ educational placement, one special day class teacher from School 2 commented that “you’re going to have two different answers …” depending on whether students attend special day classes or resources classes. “In special day class, they choose first by saying, ‘What is their strength? It is language arts, reading, or math?’ Then they say, ‘What level are they at?’ If they’re at the 2nd grade, they will take the 2nd grade level test. Generally speaking in special day class, they take only a partial test. We’re trying to make it as minimal as we can to get through.” This difference was also apparent in School 1, where students received entire state tests. A special day class teacher commented, “During the IEP meeting … the facilitator will look at the teachers for our recommendation. We usually go into those meetings after thinking that out. We’ll make a recommendation that goes to the parents. The parent hears it out, and never has a parent disagreed with our recommendation.” However, for resource class teachers, responses included, “The parent actually makes the decision.” Or, “When we have an IEP meeting, we talk with the parents about the pros and cons of out-of-level testing. The IEP team as a whole has a sense of what they would recommend, and we will tell the parent that, but we always tell the parent that ultimately it’s going to be their decision.”

In School 1, our narrative results indicated a clear pattern in teacher preferences for the level at which their students were assessed. When a student is in 7th grade, teachers appeared more open to testing further below the grade of enrollment. However, administrators said that by the time students were attending 8th grade, the need to prepare and pass the High School Exit Exam drives the decision to test a student on-grade level. “There’s a difference in 7th and 8th grade. Our 8th grade teachers want them taking it on grade level and our 7th grade teachers generally want them to take it a grade level below. We want them [8th grade students] to be compared to what they really need to know at this grade level so that we know what to work on. We know they’re going to have to do the exit exams.”

When the special education teachers in School 1 created their interview schedule to participate in our data collection process, they began discussing the selection process for testing students with disabilities out of level. Through this discussion, they learned that differences existed between the four teachers about how to select students for out-of-level tests and how to determine the appropriate out-of-level test level. Among the four teachers, two taught special education classes in language arts and two taught special education classes in math. When selecting students for an out-of-level test, language arts teachers thought about the entire test in terms of how well a student could read the language arts and the math test. One language arts instructor commented, “it [out-of-level testing] is usually based on their reading level. If they can’t read the test, they’re not going to perform well on it anyway.” Accordingly, language arts teachers tended to select test levels, again, by considering how well students could read the entire test. However, “… if the kid is doing 4th or 5th grade math, we explain to them [parents] that they’d have to take a 7th grade test. But they’re really working on 5th grade level math and it may be too difficult for them.” Both math teachers in School 1 considered students’ math abilities only in deciding to test students out of level. One math teacher, however, provided read aloud accommodations for all students for whom she administered the test “to make sure that reading abilities didn’t interfere with their performance.”
 

Table 9. Differences by Content Area of Instruction

Discussion

The final step in the interpretation of our case study data was to seek out points of commonality and corroboration between our interview data from multiple sources and our review of students’ IEPs. We discuss these points through “grand themes” that emerged from our analysis as overarching findings that we believe to be important considerations for policymakers and educators. If alternative view points emerged in our data sets, those findings are presented as caveats to our grand themes of considerations.

Students with disabilities who were tested out of level were not instructed on the grade level in which they were enrolled in school.
As a first consideration, our findings suggest that all of the students tested out of level in these two schools were not receiving standards-based instruction in all content areas that was commensurate with their grade of enrollment. Our narrative results support this conclusion in that teachers believed that some students in special education will never be able to meet grade level standards. These comments were taken from interviews with teachers who taught students whose disabilities could be considered mild or moderate since each of these students receive general education instruction for a part of each school day. In addition, both teacher-provided information about students’ reading and math achievement levels and IEP review results indicated that some middle school students were achieving at elementary grade levels. This conclusion points to two issues that are important considerations for policymakers and practitioners.

First, since NCLB mandates that all students are to receive grade-level, standards-based instruction, it is important for educators to think carefully about how to bring all students up to proficient levels of