Home | About | Publications | Special Topics | Presentations | State Policies | Accommodations Bibliography | Teleconferences | Tools | Related Sites

Where's Waldo? A Third Search for Students with Disabilities in State Accountability Reports

Technical Report 25

Published by the National Center on Educational Outcomes

Prepared by Martha L. Thurlow, J. Ruth Nelson, Ellen Teelucksingh, and James E. Ysseldyke

April 2000

Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:

Thurlow, M. L., Nelson, J. R., Teelucksingh, E., & Ysseldyke, J. E. (2000). Where's Waldo? A third search for students with disabilities in state accountability reports (Technical Report No. 25). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/TechReport25.html

Executive Summary

This report is the third analysis of state reports conducted by NCEO to determine what types of information are provided on students with disabilities in state education reports. Previous analyses had shown that few states (11 in the first analysis and 13 states in the second analysis) reported test-based results for students with disabilities. For this analysis, we collected 170 reports between March 1999 and August 1999 from state accountability offices and state special education offices.

Despite IDEA requirements for public reporting of assessment participation and performance data from students with disabilities, we found that only 14 states included participation data and only 17 states included performance data for students with disabilities in state assessments. Participation levels varied from 33% to 97% of students with disabilities. Performance levels also varied widely, with the differences between rates of students meeting state standards for students with disabilities and all students ranging from 20% to 50%. States are now required to "make available to the public, and report to the public with the same frequency and in the same detail as it reports on the assessment of nondisabled children the following information...the performance results" of students with disabilities on regular and alternate assessments (34 CFR 300.139). There were increases in the reporting of performance data for students with disabilities, but not to the extent that might be expected given the timelines in IDEA 97. More than two-thirds of the documents that did not include data on students with disabilities did include data on regular education student performance. Only eight states reported data on students with disabilities for the first time in this analysis. Furthermore, there were some unexpected reporting changes; three states that reported data on students with disabilities in previous years did not report data this time.

We present several recommendations as a result of our findings. At minimum, states should document that they recognize the need to report data on students with disabilities, and indicate why they have not yet done so and how they are moving toward meeting the IDEA requirements. The ways in which data are presented should be considered carefully so that unintended consequences do not result simply because of the way data are presented. Finally, states should be thorough in reporting—clearly accounting for the number of students actually participating in assessments and represented in every score that is presented.

Overview

When the Individuals with Disabilities Education Act (IDEA) was reauthorized in 1997, educational reform in special education was initiated at many levels. The law now requires educational results for students receiving special education services to be public information, just as they are for students without disabilities. IDEA 97 requires states to define performance goals and indicators for themselves, and one of the indicators must be the performance of students with disabilities on assessments. States must include all students with disabilities in their assessments, either in the regular assessment (with accommodations when appropriate), or in an alternate assessment. These requirements recognize that when students with disabilities are excluded from state assessments, an inaccurate picture of how students are performing may result (Thurlow, House, Boys, Scott, & Ysseldyke, 2000; Zlatos, 1994). The law also requires that the performance of students with disabilities on these measures be reported in the same detail and with the same frequency as the performance of other students is reported.

For the past three years now, the National Center on Educational Outcomes has been examining state reporting on the performance and participation of students with disabilities in statewide assessments. Due to the recent push for educational accountability for all students, including students with disabilities, it continues to be critical that we track the nature of, and changes in, state reporting practices, especially since very limited data are currently available on the results of education for these students (Thurlow, Langenfeld, Nelson, Shin & Coleman, 1998; Ysseldyke, Thurlow, Langenfeld, Nelson, Teelucksingh & Seyfarth, 1998).

Historically, states were far from meeting the current requirements. In our first study on state reporting practices, which examined state accountability reports that were collected between Fall 1995 and Spring 1997, we found that only 11 states included test-based outcome data on students with disabilities (Thurlow et al., 1998). However, a larger number of states (N=30) included process indicators for students with disabilities (e.g., enrollment, attendance) in the same reports. Recommendations for good state reporting practices were provided in this report, and many of these are reflected in provisions and regulations of IDEA 97 (e.g., performance data on students with disabilities should be publicly reported as often as data on regular education students).

Our second examination of state reporting practices did not reveal major improvements in either the number of states reporting on students with disabilities or the nature of the data provided. Only 13 states included test based outcome data on students with disabilities. Even fewer states (N=12) provided data on the participation of students with disabilities in assessments. Once again, a larger number of states (N=38) included educational process data on students with disabilities. These data that had been required for federal reporting prior to IDEA 97 include graduation and exit data, enrollment data, dropout rates, and time spent in various settings (Ysseldyke et al., 1998).

In 1998, we found that 50-80% of students with disabilities were reported to be participating in statewide assessments. Performance data indicated generally lower performance of students with disabilities compared to other students. For example, on state assessments of reading, the differences in passing rates between all students and students with disabilities ranged from 30 to 50 percentage points. It is difficult to analyze and interpret these results because of the limited number of states that reported performance and participation data on students with disabilities. However, it is important to continue to examine how students with disabilities are performing and participating in statewide assessments (Ysseldyke et al., 1998).

A survey of state directors of special education (Thompson & Thurlow, 1999) revealed that 23 states were able to provide data on the participation of students with disabilities. Yet, all states except five (four of which had not responded to the item) indicated that they disaggregated data on students with disabilities. The one state that indicated it did not disaggregate data was a state that had no statewide assessment.

The goal of our third study of state reports is to continue to track state reporting practices on the performance and participation of students with disabilities in statewide assessments. With the IDEA amendments now in force, states are federally required to report on the performance and participation of students with disabilities. At the time these reports were gathered, near the end of 1999, we would have expected that nearly all states would be reporting on the performance and participation of students with disabilities.

Method

Data were gathered from public educational accountability documents identified in the annual Council of Chief State School Officers (CCSSO) state accountability survey entitled State Education Accountability and Indicator Reports: Status of Reports across the States 1998 (CCSSO, 1998). Even when a state document appeared to contain old data (1995-96 school year data), we included it in our analysis as long as it was listed in the 1998 state accountability survey.

The contact person for each state listed in the CCSSO state accountability survey was always contacted first. These contact persons were usually in various state offices (Assessment, Accountability, Reporting and/or Special Education departments). From these contacts, we requested the documents listed by CCSSO. We were careful to also inquire about any other publications that contained any data on students with disabilities, especially disaggregated performance data. Many times we were referred to the Directors of Special Education or other knowledgeable individuals within the state special education departments for further information on reports produced on students with disabilities. We also mailed a request for reports with a copy of the previous published NCEO Technical Report 22 (Ysseldyke, Thurlow, Langenfeld, Nelson, Teelucksingh, & Seyfarth, 1998) to each state assessment director and director of special education, with the intention that this would increase the chances that we received all published data available.

Timeline of Data Collection. Data were gathered between March 1999 and August 1999. Though most reports were obtained by the end of June, many reports were downloaded off the Internet throughout the summer months. We asked for any documents published between March 1998 and March 1999, yet received a few that came after these established dates. The difficulty in collecting accountability reports is that state departments publish various documents at various points of the year. All attempts were made to include the most recent data available unless the most recent data on students with disabilities did not have the same kind of data as that provided for other students. This only occurred once. For example, New York publishes several reports at various times of the year and newer data (1997-98 test data sent to us in August, 1999) which was included in the 1999 VESID Pocketbook (University of the State of New York, New York State Education Department, & Office of Vocational and Educational Services for Individuals with Disabilities, 1999) did not provide comparison test data for all students or general education students for the Regents Competency Exams.

Inclusion Criteria of Data Analyzed. For this analysis, any published document or report that provided an author and publisher (state department) was considered to be public data that could be obtained. Internet information was considered published, public data accessible by the general population. We did not include regular education data that included scores of students with disabilities when disaggregated scores of students with disabilities were not found elsewhere in the documents. Data from special studies, grants, or projects were not included if these data were not regularly collected and publicly reported. Only state level information was included in the results (e.g., no district or school level data on students with disabilities). Data were analyzed for grades K–12, excluding any preschool or postschool data.

NCEO did receive from five states some data on students with disabilities that appeared not to be published (no publisher or authors); these appeared to be internal data reports not accessible by the general public. These were not included in the current analysis. These documents were generally database sheets or results produced by the testing company. See Table 1 for a listing of unpublished documents that did contain disaggregated data on students with disabilities, but were not included in this study.

Analysis of Data. For this analysis, we reviewed 170 public documents (see Appendix A). Each report was searched thoroughly for data on students with disabilities. A two page accountability report checklist was completed for each document on the educational results and process indicators for all students, noting carefully any disaggregated data on students with disabilities. This checklist is reproduced in Appendix B.

The data were coded according to the NCEO framework (Ysseldyke, Krentz, Elliott, Thurlow, Erickson, & Moore, 1998). This comprehensive framework, initially created by hundreds of nationally-representative stakeholders, includes both academic and nonacademic domains, and encompasses more than just participation data. The vast majority of data collected fell into only two categories: (1) Educational Results for Systems and Individuals, and (2) Educational Processes, specifically Student-Oriented Domains.

Data from three domains were collected in the area of Educational Results: Academic and Functional Literacy, Personal and Social Well-Being, and Satisfaction. Most of the data included information on Academic and Functional Literacy. Test score data included in the area of Academic and Functional Literacy were of primary interest to us, and thus were subjected to additional analysis.

Data from two domains were collected in the area of Educational Processes: Participation and Family Involvement. Most of the data reported by states in these areas are included in the Twentieth Annual Report to Congress on the Implementation of the Individuals with Disabilities Education Act (U.S. Department of Education, 1998), including enrollment, placement, exit data, and personnel and financial data. These data are not included in our analysis; however, we do mention when these data were included in public accountability reports, since data in these state reports are more widely available to the general public than the Report to Congress.

Cautions in Interpretation of the Data Analysis. Though every effort was made to gather current data, states gather and report data at different times throughout the year, and may be reporting data a year or two behind the actual date that data were collected. Therefore, it cannot be assumed that the data included here are taken from the same school year. States also use various tests, rubrics, and standards to judge performance. Data provided in the achievement data tables reflect a comparison between all students and students with disabilities if available; otherwise, the comparison groups are noted (e.g., Not Disabled, Regular Education students). As in our previous report, some data in state reports continue to be difficult to interpret because they do not include glossary or summary information, the percentages of students with disabilities tested, or other information that is needed to accurately interpret the data.

The data presented here are intended to be a general overview of the performance of students with disabilities, and should be interpreted with caution. Comparisons between states on the performance and participation of students with disabilities are not appropriate since measures and participation rates for students with disabilities vary greatly from one state to the next.

Results

Of the 165 reports that were analyzed from 50 states, a total of 91 reports (41 states) included data on students with disabilities in the domains of student-oriented process or academic and functional literacy. Of the 74 reports that did not include data on students with disabilities, over 50 included performance data on regular education students. Nine states did not have any information on students with disabilities other than financial data. Because states often produce multiple reports with varying levels of information (e.g., individual school, school district, region, and state) (see Thurlow et al., 1998; Ysseldyke, Thurlow et al., 1998), we opted to analyze all data in terms of state performance (e.g., number of states reporting on test scores or number of students participating in testing).

A listing of all reviewed documents is provided in Appendix A. In it is an accompanying legend that identifies for each state document the type of data available on students with disabilities. A summary of which states report data on educational results and processes is provided in Table 2. As indicated in the Educational Results column, the most frequent domain for which data were presented was Academic and Functional Literacy. Only three states included other areas (Kansas had Personal and Social Well-Being data; New York and Vermont had Satisfaction data on vocational and special education services as well as Academic and Functional Literacy). In the Process area, most states reported on the enrollment of students with disabilities.

Table 2. Overall Summary of Data Gathered on Students with Disabilities

Educational Results

Of the 41 states that provided some data beyond financial data on students with disabilities, 17 states disaggregated performance data for students with disabilities in the area of Academic and Functional Literacy (Connecticut, Delaware, Maryland, Massachusetts, Minnesota, Mississippi, Nevada, New Hampshire, New York, North Carolina, Rhode Island, South Carolina, South Dakota, Texas, Vermont, Virginia, West Virginia).

The 17 states provided information on how students with disabilities performed on statewide assessments. A handful of states included information in their state accountability documents on how students with disabilities are performing over time and whether there is improvement or progress in performance from year to year.

Two states reported on other domains of results for students with disabilities. Kansas, the only state to report on the area of Personal and Social Well-Being, cited the number of violent acts committed by students with disabilities against other students and teachers. New York reported data in the domain of Satisfaction: the results of a Consumer Satisfaction Survey on vocational rehabilitation services provided to special education students. Vermont administered student surveys to special education seniors that addressed student satisfaction with their education.

For the 17 states that presented information on statewide assessments, the most frequently reported content areas (see Table 3) were: reading (17 states) and math (17 states). Eleven states reported science data, ten reported writing data, and only six reported social studies data. Sixteen states reported on students with disabilities in three or more content areas (Connecticut, Delaware, Maryland, Massachusetts, Minnesota, Nevada, New Hampshire, New York, North Carolina, Rhode Island, South Carolina, South Dakota, Texas, Vermont, Virginia, West Virginia).

According to Guy, Shin, Lee and Thurlow (1999), 20 states had a high stakes graduation exit exam in 1998. All of these states reported graduation exam results for regular education students, but only 35% of the 20 states (7 states) reported these results for students with disabilities (Maryland, Minnesota, Mississippi, New York, South Carolina, Texas, Virginia).

Only a handful of states presented any other types of data in the domain of Academic and Functional Literacy. New York and Texas provided extensive data on students with disabilities in the area of Academic and Functional Literacy. These two states have state assessments in place, a graduation exam, and end-of-course assessments that include students with disabilities. Furthermore, these states have other unique indicators in this domain. New York has an Occupational Education Proficiency Exam, and Texas reports on the Texas Academic Skills Program Test (TASP). The TASP is required for entry into Texas institutions of higher education. These states also provided extensive achievement data on students with disabilities. Rhode Island reported on the unique indicator of statewide performance on a Health exam.

State achievement test data of students with disabilities can be examined in different ways. Because it is difficult to aggregate and analyze achievement data of states due to differences in tests, standards, rubrics, the time of year given, content difficulty of tests, accommodations given, exclusion of students, the grade the test was given, or the year the data were collected, we decided to examine how students performed relative to standards set by the states. We used the percentage of students above the passing score or other index of "adequate" performance. These data are presented in Table 4.

Table 4. State Achievement Test Data of Students with Disabilities (Passing State Criteria or Percentile Rank Scores)

a DE does not identify which of its four point rubric comprises an adequate or "passing" on the Writing Assessment. For purpose of display, we selected the 2.0 level as "passing." Only the percentile rank is given for the SAT/9 portion of the DTSP. DE does give standards-based scores including raw mean score, raw standard deviation, mean scale score, and mean scale standard deviation.

b The authors selected % at or above Level II as "passing" for the Minnesota Comprehensive Assessment results because the report stated that those at "Level III or above have demonstrated more than simple minimum competency" (p. 29, Office of Educational Accountability, 1999).

c The authors selected 1996-97 RCT data for NY because data for all students were available as well. More current data (1997-98) of the performance of SWD for the RCT were available, but no data were available for all students for the same academic year (1997-98). 1997-98 data were available for both SWD and all students for the Regents Exams, and thus, were included. Data were only included if comparisons could be made. Finally, results on SWD for the Regents Preliminary Competency Tests were available, but not for all students; thus, these data were not included.

d NC provides disaggregated results by category of special education services for the math and reading end-of grade and end-of-course tests. We have calculated the aggregate data for students with disabilities from information provided in their state reports. ELP refers to Economic, Legal & Political Systems EOC test.

e The authors chose the 51 – 99 percentile range as the passing score. SC did not specify a passing criterion, but presented the percentages by national quarters.

f SD provides disaggregated scores for students on an IEP who took the test under standard conditions (IEP St), standard accommodations (IEP Accom), and with nonstandard (NS) accommodations (IEP NC Accom) as well as for students on a 504 plan who took the SAT/9 under the same three conditions.

g VT provided bar graphs of the disaggregated results, and so we had to approximate the percentage passing.

h VA also disaggregates results of the SAT/9 by category of students receiving special education services. Not enough information is available to calculate overall scores for SWD in each content area. There is also not a passing descriptor for the SAT/9.

i WV tests in these areas as well as SAT/9 Language, Spelling, Study Skills, and Listening areas. However, data disaggregated for SWD are not given by content area, but for the average of all students with disabilities (excluding gifted) tested under standard conditions in grades 3 – 11 on the total basic skills scores (does not include SWD who take the test under nonstandard conditions). We chose the 51 – 99 percentile range as the passing score. WV did not specify a passing criterion, but presented the percentages by quartiles.

Reading Achievement. Figures 1 and 2 depict the differences between percentages of all students and students with disabilities meeting standards on criterion-referenced assessments in reading. These figures are based on data in Table 4. As indicated, the performance of students with disabilities was quite variable compared to all students’ performance. There were not only differences across states, but also within states on different assessments (perhaps reflecting the difficulty of each assessment). From the data reported, 5% to 87% of students with disabilities tested are meeting reading standards (see Table 5). In fact, these divergent passing rates came from different tests in the same state (Maryland). The differences in passing rates between all students and students with disabilities ranged from 12 to 49 percentage points (see Figure 1). Maryland and New York had the smallest gap (12 and 16 percentage points) in reading performance on their functional or competency exams (Maryland’s Functional Tests and New York’s Regents Competency Exams). In grade 8, the differences in the passing rates between all students and students with disabilities in reading performance ranged from 23 to 47 percentage points (see Table 5 and Figure 2).

Figure 1. CRT Reading Differences Between Students With and Without Disabilities

Figure 2. CRT Reading Differences Between Students With and Without Disabilities (Grade 8)

As expected, when examining the reading differences between all students and students with disabilities on norm-referenced tests (NRT) in Figure 3, the performance was not as variable. Students with disabilities from the five states that presented NRT data performed between the 13th and 26th percentile (see Table 5). The percentage differences in reading performance between students with disabilities and all students spanned 26% to 43%. When looking at the same NRT (SAT/9) reading assessment used by both Delaware and South Dakota (refer to Figure 3), the difference in performance results may be a reflection of the method of reporting procedures. South Dakota disaggregates scores by students with disabilities on an IEP taking the assessment under standard conditions, those taking it with standard accommodations, and those taking it with nonstandard accommodations. In Figure 3, we used the percentage of those on an IEP taking the assessment under standard conditions since an aggregate score for all students with disabilities tested was not given. Delaware, on the other hand, does not make that distinction, and it is unclear who is included in their percentages (e.g., all students on an IEP using standard and nonstandard accommodations, all students on an IEP tested under standard conditions or using standard accommodations, or just students on an IEP testing under standard conditions).

Figure 3. NRT Reading Differences Between Students With and Without Disabilities

Table 5. Reading Achievement Differences Between Students With and Without Disabilities

Math Achievement. Performance in math was just as variable as performance in reading. From all data reported, 3% to 74% of students with disabilities are meeting math standards. The differences in passing rates of students with disabilities and all students ranged from 13 to 42 percentage points (see Table 6 and Figure 4). As detailed in Table 6 and visually depicted in Figure 5, achievement differences on criterion-referenced exams in 8th grade between all students and students with disabilities ranged from 19 (VA Literacy Passport Test) to 42 percentage points (MN Basic Standards Test and NC End-of-Grade Test). Maryland and New York, again, had the smallest achievement differences (e.g., 16 and 13 percentage points) between these two groups of students.

Students with disabilities from the five states that presented NRT data performed between the 15th and 33rd percentile (see Figure 6). The percentage differences between all students and students with disabilities in math achievement were very small across these five states: 28% to 37%. Again, South Dakota’s passing percentages for the same SAT/9 math assessment as compared to Delaware’s SAT/9 results appear to be much larger, but we used the percentage of students on an IEP under standard conditions since there was no aggregate score for all students with disabilities’ performance.

Table 6. Math Achievement Differences Between Students With and Without Disabilities