2. Type of Assessment.   The specific type of assessment under consideration is another factor that should be listed and considered further when trying to gain greater participation of all students in an assessment system.  The type of assessment will have implications for the use of accommodations.  Among the most common types are:

These can be categorized according to the type of response requested, such as closed-ended response items and open-ended response items.

You can also describe the type of assessment in terms of the basis for scoring. For example:

• Is the assessment norm-referenced?  If it is,

  • Is it norm-referenced on a population outside of the state?

  • Are state norms used?

  • Were students with disabilities included in the norming sample?

• Is the assessment scored against an absolute standard or rubric*?  If it is,

  • Were students with disabilities included during instrument development?

  • Were students with disabilities considered when standards for performance were set?

*  Rubrics indicate degrees to which an absolute standard has been met.  For example, Kentucky uses the terms Novice, Apprentice, Proficient, and Distinguished in its rubric.

Descriptive information will help you set the framework to revise the policies and procedures in your state assessment system.

3. Purpose of Assessment.  Not all assessments in a state system serve the same purpose. Different assessment purposes may have different implications for policies and procedures. Many states have systems that include multiple assessments. Often, the different assessments are used for multiple purposes. This means that you will need to take time to examine each component of your assessment system.

Common purposes of state assessment systems are to:

• Describe student competence to inform the public

• Make comparisons among educational units (districts, schools)

• Achieve system accountability (evaluate the extent to which standards are met)

• Set policy based on student data

• Make decisions that affect student progress (minimum competency tests, grade or course promotion tests, high school graduation exams)

• Make decisions that affect the employment of school personnel (extent of student progress determines teacher salaries, school leadership)

The first four purposes are considered to be low stakes, while the remaining two are considered to be high-stakes. A low-stakes assessment has no consequences for a particular group or individual within the group. A high-stakes assessment has consequences for a particular group or individual within the group.

A test that determines whether a student will graduate is considered to have high stakes for the student. When improved student test scores in a district determine whether the superintendent will be rehired, that test is said to be high stakes for the superintendent, but not necessarily for the students. Usually, a reported test that does not affect anyone (i.e., there are no rewards or sanctions) is considered to be low stakes.

You may find that reporting results creates higher stakes than before. Statewide "report cards" on schools and school districts have tended to affect real estate values, public image, and local school board and superintendent tenure. These consequences go beyond their purported "lower-stakes" purposes. For example, when the newspaper reports poor results for a district, parents may call for the resignation of the superintendent. Low stakes and high stakes are relative terms. Yet, they help in considering the intended and unintended consequences of various assessment policies.

If you use an assessment for high-stakes purposes, be sure to document and communicate the exact consequences of the assessment and how they are applied. Some of the questions you should answer are:

• Are the "high stakes" of the assessment directed to a local education agency, a school, an administrator, a teacher, or a student?

• Is a "warning" used to allow the subject of the high stakes to make changes before consequences are imposed (e.g., a school administrator is alerted to develop a restructuring plan or face a takeover by the district or state)?  (Note: This is a legal requirement)

• Is an assistance program available to assist in remediation efforts?

• Will the high stakes have unintended consequences?   What are they?

• How are assessment consequences currently applied to students with disabilities?

Often, the consequence of exclusion is to make a district or an administrator look better (as when low-performing students who have cognitive disabilities are excluded from assessments designed to be used in making merit decisions for administrators). There is no intended consequence for individual students, yet there is a consequence for them.

Identify the Philosophies and Attitudes that Drive Your Assessment System

To revise your state assessment system, you must build on a foundation of philosophies and attitudes that recognize the need to be accountable for all students. It is essential that you identify the assumptions and philosophy under which the current assessment system operates and to determine the extent to which these support an assessment that includes all students. Working to identify the assumptions of your revised assessment system is a primary focus of Step 2.

A state's assessment system generally is constructed to reflect the state's goals for its students or the state's curriculum framework. You need to evaluate the extent to which your state's curriculum framework and standards reflect the curricula for all students, including students whose educational program emphasizes life-role skills. If you look at other states, you'll find that several have defined goals in core academic areas (see the Delaware example), whereas others have established life-role goals (see the Kentucky example).

An assessment system that focuses only on academic skills represents a greater challenge to the participation of all students than does one that addresses the educational needs of all students, including those with more severe cognitive challenges. Academically-focused goals, however, should never be used as an excuse for a state assessment system that fails to promote the participation of all students.

Summary

The first step in evaluating and revising your state assessment policies is among the most important. By involving stakeholders up front, you increase the probability that your revised state assessment will be relevant to their needs and acceptable to them. you'll need to conduct periodic revisions and improvements. As you revise, remember to continue taking the time to really look at what the current status of your assessment system is, and to look at its effects (intended and unintended). By doing this, you will have a much better foundation for addressing the key issues as you work to improve your system.

Before you proceed to Step 2, take an inventory of what you and your stakeholders have shared. What is the assessment system like? Do you know when and to whom the assessment is administered? Do you know the types of assessments that are used? Do you know the purpose of each component of your state assessment system? Is the assessment considered to be high stakes or low stakes, and for whom?

Do you think that your assessment system was designed for all students in your system? Are stakeholders in agreement that an assessment system should provide accountability for all students in the system? Are all parts of the accountability system or assessment appropriate for all students?

Step 2

Decide What You Want To Do

Your second step in revising state assessment guidelines is to agree on the general plans for revisions, define the assumptions that will underlie the assessment, and develop plans for making changes in guidelines.

You must involve stakeholders in this second step. You may want to add to the team used in Step 1. In this step, it is particularly important that you include parents and teachers. It is advisable that you involve individuals who can go back to a larger group of similar individuals and share the assumptions and plans developed in Step 2. Therefore, it is sometimes helpful to select official representatives of groups who have access to the boards, newsletters, and meetings of the larger group of stakeholders.

It also is essential that you devote sufficient time to Step 2. Without enough time to thoroughly involve key stakeholders, to spend time hashing out assumptions, and to make plans, you may jeopardize all other steps.

Agree on General Plans for Revisions and Identify Available Resources

Before you start a revision process, know generally what you want to do. This means you need to agree on your general goals and what resources you will have available to use in making revisions.

Goal of a revised assessment system. The specific goal that you set within the general framework of increasing the participation of students with disabilities in your state assessment can take many forms. Some of the possibilities are:

• Increase participation in the current system:

Use the same assessment procedures as are now used, but change the participation guidelines, the accommodations guidelines, the reporting procedures, or any combination of the three.

• Expand the system:

Add a new form of assessment that will be appropriate for studnets with different educational goals (e.g., students with more severe disabilities).

• Revise the system:

Completely revise the entire assessment system.

Be sure you have agreement on what your revision goal is before you start. Among the questions you should ask are:

• Does your goal call for development of new systems or only revisions of old systems?

• Are there portions of the system that must be revised and others that might have lower priority for revision?

Identify available resources. You must garner the resources available to you for the revision process. Identify people resources, equipment resources, and knowledge resources. Relevant questions for you to answer include the following:

• What existing stakeholder groups might support an effort to revise the assessment guidelines?

• What accommodations already exist in the assessment system (e.g., Braille, large print, audio tapes, videos of instructions in American Sigh Language, etc.)?

• What equipment or resources exist in your state to help with assessment accommodations (e.g., regional SEA offices, agencies with Brailling or enlarging equipment, distribution centers for technology/assistive devices, pools of signers or readers)?

• What sources of funds are available that could be tapped to assist in the revision process?

• Who, or what agencies, in your state have particular expertise or access to knowledge that could help with the decision and improvement process?

You should take advantage of the experiences and efforts of other states. The National Center on Educational Outcomes (NCEO) has compiled a set of state guidelines, both for participation in assessments and assessment accommodations. In addition, NCEO has produced a number of reports on these issues (see Resources as the end of this document). By reviewing these materials and contacting a few states, you will broaden your perspective on the options available to you.

Define Your Assumptions

It is important for you to state explicitly the assumptions that will underlie your revised assessment system. To help you and your stakeholders do this, a set of possible assumptions is provided below, together with a brief explanation of the reasons for each assumption. Using Worksheet 2, your team (and probably other stakeholders) should determine to what extent you agree with the assumptions. Then change them as needed, and add new ones that are identified.

Plan Your Approach

After a set of assumptions has been agreed upon, you need to plan for development and revisions. You may wish to form another advisory committee at this time. This group would help you consider all of the ramifications of your approach.

You will have to decide what assistance you need in order to proceed. It could come either from within your own agency or from the outside (e.g., University-based personnel within the state, a technical assistance center, or external contractors). If you choose to use external contractors, you should consider developing a request for proposals and conducting a proposal review process to select a group to help develop the guidelines or alternative assessments you might need. If you have an ongoing contract with someone for the overall assessment, it might be best to build greater participation of students with disabilities and accommodation developments into the contract requirements.

A simple format for documenting your plan is outlined in Worksheet 3. This format is based on the steps in this self-study guide. You also may wish to insert additional steps and delete others after you have completed all eight steps in this guide. You might want to chart where tasks overlap and which tasks relate to or depend on the prior completion of other tasks.

You will find it most helpful to go through the following sequence when documenting your plan:

1. Outline the steps/stages in the process, perhaps starting from the last step and developing your plan backwards.   Integrate your steps with milestones in the overall plan.

2. Project how the timelines in the overall plan might relate to your steps.  Starting from the last step, determine when the steps must be completed, how long they will take, and when they must start.

3. Determine who must be involved in each step and assign an individual to lead the step (even if a team is involved).

4. Estimate the resources needed for each step and insert additional steps as needed to ensure that the resources are obtained.

5. Revise the plan as necessary to reflect the realities of time, personnel, and resources.

Whatever sequence you use when developing your plan, it is important to document it and share it with others to obtain their feedback.

Summary

In Step 2, you are deciding what you want to do. After agreeing on general plans and identifying resources, you are ready to define the specific assumptions upon which your revised assessment system will be based. After this is done, and you have obtained broader feedback on the assumptions, you can set your plans for pursuing revisions and improvements. Once again, it is important to involve stakeholders in all of these processes.

Step 3

Develop Guidelines About Participation in Assessments

Your third step in revising state assessment guidelines is to agree on the words to write about participation. Your words must reflect the assumptions and goals established in Step 2Ñwords that become part of your guidelines for assessment of students with disabilities. In order to generate words that are understood and acceptable, you must plan to review and evaluate your goal and guidelines.

Remember stakeholder involvement is essential. Take care to ensure that all critical perspectives are represented (e.g., assessment, disability, local, state). Many more individuals and agencies may have an investment in the state assessment system than originally thought by those involved in revising the system.

If your state has more than one assessment, consider each separately. In most cases, the different assessments are used for different purposes. When considering each assessment, take into account its purpose.

Examine Your Current Written Guidelines and Evidence of Actual Practice

To examine who participates in your state assessment system, you need to examine both written guidelines and evidence of actual practice.

Most states now have some existing written guidelines that address the participation of students with disabilities in assessments. Many times these are combined with guidelines on the participation of students who are learning English (variously referred to as English language learners, students with limited English proficiencyÑLEP, English as a Second Language (ESL) students, and other terms). It is probably best to address students with disabilities separate from students with limited English proficiency.

The location and exact wording of the current guidelines should be made available for further reference. Some states include their guidelines in statutes and others have them in regulations. However, most states have them in separate, non-legal documents.

States with Written Guidelines on the Participation of Students
with Disabilities in Statewide Assessments (1995)

There are a number of ways to obtain evidence of the actual participation of students with disabilities in assessments. First, you should ask several individuals (WHO?) questions such as:

• To what extent are local school personnel aware for the guidelines?

• What is the general attitude about the guidelines among SEA staff, local assessment personnel, special educators, and parents?

• To what extent do people say they follow the guidelines?

• How is implementation of the guidelines checked?

Next, you need to look at assessment data to assess the actual participation of students with disabilities in the state assessment. Questions that should be addressed include:

• How are samples drawn and do they include students in separate schools and students in separate classes?

• Is there documentation of exemptions?

• Do rates of nonparticipation in an assessment differ for students with different characteristics?

• What percentage of students with disabilities in the state have data that you can use?

These kinds of information may not be part of your state database. You might have to do a considerable amount of searching to find them.

An even better way for you to document participation, of course, is to conduct a study in a sample of schools to determine actual participation rates. If this is unreasonable for your state to pursue at this time, you can get a good estimate by talking to many people, or by surveying or interviewing key respondents.

Review Your Goal for Improving the Assessment System

Before proceeding further, you need to review and restate your goal for improving the assessment and the assumptions that underlie the new system. Remember, the goal that you decided upon in Step 2 was one of the following:

• Use the same assessment procedures, but change the participation guidelines

• Add a new form of assessment that will be appropriate for students with different educational goals

• Completely revise the assessment system

Write Specific Guidelines

The specific guidelines that you write will vary somewhat with the original goal that was identified. You can consider the following possible approaches:

Use the same assessment procedures, but change the participation guidelines. Many states can immediately increase the participation rate of students with disabilities beyond the 50% level simply by changing written guidelines (and ensuring that they are followed). The possible ways in which this could be done might be identified through a brainstorming session involving stakeholders. Some example guidelines are:

• Use a team to decide on the participation of a student only if the team has received training in the importance of including all students.

• Require the decision maker to document the reasons for exclusion of any student.  And, ensure that these reasons are examined for appropriateness.

• Provide a checklist of considerations in making the decision about participation in the assessment.

• Require documentation of the number of students excluded for various reasons.

• If an assessment is a high stakes assessment (such as a graduation exam), require the student and the student's parents to sign off on a form giving the reasons for exclusion and the consequences of nonparticipation (such as not receiving a reular diploma).

• Examine and remove words that provide a reward to a student for not participating in an assessment (such as when any student given an exemption from an assessment automatically receives a regular diploma).

Linked with the notion of changing participation guidelines is the notion of accommodations. You will find these discussed in further detail as part of Step 4.

Add a new form of assessment for students with different educational goals. Depending on the nature of the primary assessment, your guidelines might need to identify students who should be exempted from the regular assessment. In such cases, you will want to look at a new form of assessment for these students. The new assessment form should be developed only for those students who are not working on the same kind of goals as are other students. For example, students with more severe cognitive disabilities may have educational plans that target instruction on self-help and independent living skills. These goals may be related to reading in that they include the discrimination of key signs and symbols in the environment. For example, an assessment that checks recognition of words like "restroom," "men," "women," "stop," "elevator," "exit," and "information," along with a variety of symbols could be a new assessment to complement the state's reading assessment.

Assessments can be developed in a number of ways. Kentucky wrote a subcontract for the development of its alternative portfolio system as a part of its contract with the firm developing the overall assessment system. That firm, in turn, worked with a project at the University of Kentucky to develop and pilot-test the system, train scorers, and operate the system for the first few years.

Michigan funded a project through a private corporation in order to develop its sets of outcomes and the related assessment procedures. This firm, which specialized in disability research, drafted the instruments and conducted extensive studies to check feasibility and validity.

The American Institutes of Research (AIR) developed an instrument called the Performance Assessment for Self-Sufficiency (PASS), which uses an informed respondent. All work was conducted under a contract with the U.S. Office of Special Education Programs.

Developing assessments via a contract might take you one or two years. In the interim, you could use an alternative assessment, such as parent and teacher reports on an existing adaptive behavior measure.

It is necessary that your criteria for deciding when a student's goals differ from those of the regular curriculum, and thus that the student should be assessed differently, should be clear and stringent. It would be unwise (and even unethical) to place a student in a functional living skills curriculum solely for the purpose of allowing the student to participate in the different assessment system rather than the regular assessment system. A possible approach to determine which students should participate in a different assessment might be to develop a checklist of characteristics. Students who pass criteria, such as those outlined here, would then be assessed using another measure that produces statewide data.

Completely revise the assessment system. It was expected that as states began to develop new performance assessments, they would do so in a way that made the assessments truly appropriate for all students. This has not happened. But, the idea of completely starting over is still reasonable for some states. For example, many states have recently revised their curriculum frameworks and are in the process of making major changes in their state assessmentsÑpresenting an opportunity to be more inclusive. Starting from scratch probably is the best way to create an assessment system that really includes all students.