Krentz, J., Thurlow, M., & Callender, S. (2000). Accountability systems and counting students with disabilities (TechnicalReport 29). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Technical29.htm

This report is the first by the National Center on Educational Outcomes (NCEO) to examine the extent to which students with disabilities are included in accountability systems of states in which there are high stakes for educational systems. Conducted through a Web search of states’ education Web sites and direct links to their Web sites, we examined the indicators on which states collect information (e.g., student indicators, staff indicators, program indicators, and use of resources indicators) as well as the consequences provided by the states (e.g., rewards, assistance, or sanctions).

Of the 280 indicators listed for these states according to the Education Commission of the States (ECS), NCEO was able to find documentation of 178 of them on the states’ Web sites or direct links. Furthermore, NCEO found an additional 216 indicators not identified by ECS. However, it was difficult to determine the extent to which any of these indicators were used in making determinations for consequences. Information for many indicators was collected and reported but not included in the actual formulas for determining rewards, assistance, or sanctions. Most often, assessment scores served as the only indicator used for making this determination.

Five states had accountability formulas that could be identified in information found on state Web sites. Louisiana, Mississippi, New York, North Carolina, and Ohio listed their indicators and criteria as well as how they are used to determine the consequences for states. From the information found, other states may have some of these components of an accountability system, but did not provide the level of detail on their Web sites needed to determine how their systems were implemented.

This study highlighted that not only was it difficult to determine exactly what indicators factored into overall accountability systems, but it was also difficult to find evidence that students with disabilities were included in accountability systems. Although students with disabilities may be included in assessments, their scores may not be included in the accountability systems, and most states did not specifically state whether students with disabilities were included when defining the indicators used.

Overview

Educational accountability is gaining attention and influence as state and local policymakers implement sweeping accountability reforms. States are increasingly relying on large-scale assessments to measure student performance and to determine sanctions and rewards (Education Commission of the States, 1999; Olson, Bond, & Andrews, 1999). The Web sites of major organizations, such as the Council of Chief State School Officers (CCSSO: http://www.ccsso.org) and the Education Commission of the States (http://www.ecs.org) reflect this emphasis in content on their Web pages and their on-line reports.

Approximately half the states now use, or will soon use, tests in making decisions about student promotion (see Quenemoen, Lehr, Thurlow, Thompson, & Bolt, 2000) and high school graduation (see Guy, Shin, Lee, & Thurlow, 1999). All but two states have statewide assessments in place, many with more than one assessment (Olson, Bond, & Andrews, 1999). In the 2000 legislative sessions, over 300 bills were introduced in 41 states that dealt with statewide assessments (Coleman, Heller, & McNeil, 2000; see also the Web site for the National Conference of State Legislatures, http://www.ncsl.org/programs/educ/k12link.htm).

Not only are states increasingly attaching high stakes consequences to students for their performance on assessments (e.g., no diploma, no promotion to the next grade), but states are also attaching significant consequences to various aspects of the educational system (e.g., to teachers, schools, and administrators). This approach, known as system accountability, occurs when educators, administrators, schools, or districts are held responsible and consequences are assigned to them for various student performance indicators. For example, school accreditation may be based on test scores; teachers may receive cash rewards for student performance; or, administrators may be either sanctioned or rewarded based on school improvement, which in turn is measured to a large extent by student performance.

Within system accountability, there are different levels of consequences – referred to as the “stakes” of the assessment. At one end is simply publishing test scores – usually considered to be low stakes. While it might be considered “positive feedback” to have a school that performed well on test scores, or conversely, might prove an embarrassment for schools or districts that performed poorly, these consequences are relatively minor. In contrast, if teachers are assigned to “less prestigious” schools or receive cash bonuses based on student performance, the stakes are considerably higher. Similarly, if principals receive increased school funds or lose their jobs based on how students in the school perform, the stakes are very high.

In addition to obvious “rewards” and “punishments,” there is another type of consequence currently used in accountability systems. This type of consequence involves assistance to the system and is generally considered to be a much less significant consequence.  An example of assistance might include the provision of additional resources, such as monetary grants or specially trained personnel to schools that receive a “warning.” The distinction between “assistance” and “sanctions” is often subtle, at best. The “assistance” phase sometimes serves as a probationary period prior to more intense intervention or reconstitution of a school or district.

The Education Commission of the States (1999) has documented the specific indicators that states are using in their accountability systems. Although an array of indicators is listed for most states, it is unclear how they are used. Many questions remain about current and proposed accountability systems. Which indicators are truly high stakes? Which are simply included in a “report card” for a school or district? Are the indicators weighted in some manner or incorporated in a formula to develop an accountability measure? Are there positive rewards for successful schools and systems, assistance for struggling schools and systems, or sanctions for those who are not performing as expected?

In addition to these basic questions about state accountability systems, there are also questions that focus on specific groups of students. With the reauthorization of the Individuals with Disabilities Education Act in 1997 (IDEA 97), states and districts must include students with disabilities in their assessments. They are also required to report on the performance of students with disabilities, both aggregated with other students, and disaggregated.

Several analyses from before and after the enactment of IDEA 97 indicated that states have started to include students with disabilities in their assessments only recently (McGrew, Thurlow, Shriner, & Spiegel, 1992; Thompson & Thurlow, 1999), and that many states are not yet reporting the results (Thurlow, Langenfeld, Nelson, Shin, & Coleman, 1998; Thurlow, Nelson, Teelucksingh, & Ysseldyke, 2000) even though they are required to do so by federal law. It is a logical follow-up question to ask about the extent to which the increasing participation of students with disabilities in assessments translates into their inclusion in accountability systems.

Of course, there are many reasons for including students with disabilities in accountability systems (Thurlow, Elliott, & Ysseldyke, 1998). If these students are not included in accountability systems, it means that roughly 10% of the student population is excluded from accountability. Students who are left out of assessments and accountability systems tend to not be considered when reform efforts are being designed and implemented. Furthermore, the education system does not obtain an accurate picture of education if a significant portion of students is excluded, nor is it able to make “fair” comparisons. Participation of students in accountability systems seems to promote higher expectations for those students who are included, and consequently, may be assumed to exclude from high expectations those students who are not included.

The purpose of this research was to examine the extent to which students with disabilities are included in accountability indicators in those states in which there are high stakes for systems. To do this, we had to first decipher the accountability systems that states use – this involved identifying both the indicators that are part of the accountability system and the “formula” that is used in each state to combine the indicators to determine an accountability score. We restricted our search to accountability indicators and formula used to determine system-level consequences and did not search for formula developed exclusively to meet Title I requirements for adequate yearly progress. From the indicators and formula identified, we examined the extent to which students with disabilities were included in the state’s accountability system.

Method

Initially, we used the document Education Accountability Systems in 50 States (Education Commission of the States/ECS, 1999) to determine which states we would include in our analysis. We selected states that ECS had listed as having a “statute” or “regulation” in the Rewards or Sanctions column of its Table I. There were 38 states that met this criterion. The ECS document also included information about which indicators were used by the various states (see ECS Table 5). We divided in half the list of states that had high stakes accountability systems for schools or districts, then two of us examined Web sites to verify and expand on the information that was included in the ECS report.

We used the Achieve Web site (http://www.achieve.org) “State Links” page as a quick link to the Web site for the Department of Education of each state. From this site, we also located other state offices (e.g., accountability, assessment, legislature).

We also used the Council of Chief State School Officers (CCSSO) Web site (http://www.ccsso.org) section on “Council Projects” where we located the Accountability Systems Profiles. (The direct Web address is http://www.ccsso.org/introprofile.html). According to this site, the Resource Center on Educational Equity of the Council of Chief State School Officers (CCSSO), in collaboration with the New England Comprehensive Assistance Center, is developing profiles of several state’s education accountability systems.

When we checked the site, there was information about the following 13 states: Alabama, Connecticut, Georgia, Illinois, Kentucky, Maryland, Massachusetts, New Jersey, North Carolina, Oregon, Rhode Island, Texas, and Vermont. The profiles included information about each state’s content and performance standards, statewide student assessment, and criteria used to determine whether schools or districts are eligible for rewards or are in need of improvement. The site also contained information about how school and district performance is publicized, as well as details about the provision of rewards, assistance, and sanctions. There was often helpful information about the inclusion of students with disabilities in the assessment systems, as well as information about provisions for English Language Learners.

The Resource Center on Educational Equity page of the CCSSO Web site indicated that the information for the 13 profiled states was gathered from various sources, including CCSSO reports, state statutes and regulations, and information posted on state education agency Web sites. The Center’s initial drafts were reviewed and revised by state education agency staff. We used the information from CCSSO’s state profiles only after independently verifying it on state Web sites.

At each state’s Web site we reviewed state accountability reports, report cards, policy guidelines, and other relevant items to identify indicators, consequences, and accountability formula. This review was conducted in 1999-2000, with all reviews completed by July 2000. After we completed our initial examinations of Web site information on states, the two lists of states were traded, so that the information could be verified. In some instances, there was little change from what the first examination had indicated. In other cases there was completely new and sometimes contradictory information obtained during the second examination of the Web site. This report contains the more recent version of those sites.

Results

It became very evident during our verification process that the information we sought was a moving target. It changed rapidly due to ongoing legislative processes, subsequent rule-making procedures that follow legislative sessions (in order to implement the legislation just passed), and states’ action taken in response to federal legislation. Our focus was on Web-based information, and we found a significant disparity between the level of sophistication and quality of information found on various states’ Web sites. Some seemed to have frequent additions or modifications, and others seemed to be fairly rudimentary. Links to other Web sites were prominent and user-friendly on some sites, and unavailable or very obscure on others; sites were frequently “down” or “temporarily under construction.” For example, important documents found during the first review were often moved or unavailable during the second review. Sometimes Web sites would change dramatically within a few days.

We collected information on the indicators used for accountability purposes, the formulas for calculation of school or district scores for accountability purposes, the consequences of the accountability systems (e.g., rewards, assistance, or sanctions), and the inclusion of students with disabilities in the accountability systems’ indicators and calculations.

Accountability Indicators

Table 1 summarizes the numbers of indicators that were identified in the ECS document and in our Web search. Of the 280 specific indicators listed in the ECS document, we were able to verify 178 of them when examining the State Departments of Education Web sites and direct link sites. In other words, we found, on average, only 64% of the indicators that ECS found using written reports. (Values ranged from 15% for Massachusetts to 100% for Alaska, Colorado, Kansas, Kentucky, New Jersey, Oklahoma, Rhode Island, South Carolina, Tennessee, Vermont, and West Virginia.)

Table 1. Indicators in State Accountability Systems