October 2003

State assessment programs provide one method of determining student academic achievement. The Washington State Assessment System (WAAS) provides accountability for program and educational opportunities for all students. Alternate assessment for students with significant cognitive disabilities, as part of Washington’s assessment program, ensures a unified system, program, and student accountability linked to the common core of learning within the general curriculum. The Washington Alternate Assessment System process was developed by the Washington Alternate Assessment Task Force and expanded by Advisory Panels in response to the following requirement in the Individuals with Disabilities Education Act 1997 (IDEA): "The State has established goals for the performance of children with disabilities in the state that . . . are consistent, to the maximum extent appropriate, with other goals and standards for children established by the state." It was toward fulfillment of this requirement that alternate assessments are based on Washington’s Essential Academic Learning Requirements (EALRs) in the content areas of Communication, Reading, Writing, Mathematics, and Science. The Washington Alternate Assessment System is a portfolio assessment designed for a very small percentage of the total school population for whom traditional assessments, even with accommodations, would be an inappropriate measure of progress.

The basic building block of the portfolio assessment is evidence of the student’s work. Each of the entries in the portfolio documents two dimensions of learning: progress on IEP skills linked to the EALRs and student generalization of those skills. Evidence of the student’s work demonstrates participation in and progress toward those IEP goals that are aligned to state standards (EALRs). In this way, evidence of progress on IEP skills linked to the EALRs can measure progress on state goals and standards. Student generalization of skills evidence shows the extent to which a student can demonstrate the IEP skill linked to EALRs in the following ways:

• Use of appropriate modifications/adaptations, supports, or assistive technology to demonstrate all the student knows and is able to do.
• A variety of settings and contexts in which the student is able to use learned skills. These places can include the classroom, other areas of the school, community settings, and home.
• Interaction with nondisabled peers and others during IEP activities for the purpose of developing social relationships to enrich his or her life.
• Use of self-determination skills in planning, monitoring, and evaluating IEP skill activities.

The inclusion of students with disabilities in the assessment and accountability system is critical to ensure appropriate allocation of resources and learning opportunities for these students. Federal legislation and regulations for the Elementary and Secondary Education Act (ESEA) and IDEA require states to report results for all students assessed using general assessments and alternate assessments relative to the same grade level academic content and achievement standards. In anticipation of the federal government publication of a Notice for Proposed Rule Making to allow setting alternate achievement standards for students with the most significant cognitive disabilities who participate in alternate assessments, the Office of Superintendent of Public Instruction sought to establish four levels of performance based on alternate achievement standards on the WAAS assessments in the fall of 2002.

The Special Education Assessment Advisory Panel reviewed the standard setting methodology used by three states, and reviewed synthesis reports and policy directives from the National Center on Educational Outcomes (NCEO). The advisory panel also reviewed the types of scores generated for the portfolio alternate assessment.

The test contractor worked with staff of the Office of Superintendent of Public Instruction (OSPI) to develop a set of procedures for standard setting for the WAAS portfolios. The Washington National Technical Advisory Committee (TAC) reviewed and made recommendations relative to these procedures, and the Washington Special Education Assessment Advisory Panel reviewed and commented on these procedures

To set standards on the WAAS portfolio assessment, a variation on a holistic method of standard setting that the contractor had used in the past was recommended to the panel and the TAC. The method is loosely based on Jaeger and Mills’ method (Cizek, 2001). With this method, the panelists’ task is to classify student work into one of several performance categories defined to capture levels of performance as expressed by the performance-level categories. The panelists review the full range of scoring patterns and are asked to sort these score profiles into four performance levels. The method is holistic in that the panelists consider the whole of an individual student’s work. The panelists then review folders of student portfolios sampled to represent the full range of scores, and are asked to sort these folders into four performance levels as represented by the quality of the students’ work.

With this WAAS portfolio standard setting method, panelists review the implications of their standards in the form of impact data. Panelists receive cumulative frequency distributions of student scores that allow them to see the percent and number of students in each category given the standards the group of panelists has set. These data are made available to panelists after they complete the two sorting processes.

The procedures use these standard-setting methodologies and a consideration of standard-setting principles in order to optimize the efficacy of this process. The goal of the standard setting is to recommend performance thresholds or cut scores for the WAAS portfolio assessment that have been established by Washington educators, subject matter experts, and administrators in the best interests of students and the overall educational process. For each subject matter, the final achievement category is decided from a minimum subject matter rating and a minimum combination of ratings on generalization skills. The object of the standard-setting process is to decide how to combine the ratings so that the portfolios can be placed in four achievement categories. The object is not to come up with a new rubric or to change student scores but to find a way to combine them so that these students can be included in reports and evaluations.

The portfolio standard-setting meeting was facilitated primarily by the contractor. OSPI staff explained the ESEA and IDEA background that is the impetus for standards, the history of the WAAS portfolio, and presented the academic achievement standard descriptions. An OSPI psychometrician documented the process to ensure adherence with the planned steps for standard setting.

The results of standard-setting for the portfolios were shared with the Special Education Assessment Advisory Panel and with the National Technical Advisory Committee, and results of standard-setting were then presented to the Superintendent of Public Instruction.

Selection of Standard Setting Panelists

Table 1 indicates the composition of the 14-member standard setting panel that was selected to set the standards for the Portfolio. Each person participating in the standard-setting process was selected for his or her qualifications as a judge of student performance based on various factors. Teachers, educators, and subject area experts who were selected as panelists exemplified the required subject-area knowledge, teaching experience, or understanding of students necessary for an appropriate and comprehensive standard-setting study. Each panelist participating in the process represented the knowledge and understanding of his or her peers throughout the course of the process, lending a balance between diverse opinion and consensus. To ensure that a diverse opinion was obtained, some of the participants had not participated in making recommendations during the development of the portfolio system or in the scoring of the portfolios.

To ensure balance, a stratified sample of school district staff and other stakeholders throughout Washington were contacted. A concerted effort was made to balance the panel based on county representation, urban representation, representation of students with significant disabilities and schools serving various sizes of populations, gender, and race/ethnicity. The overall goal of consensus in this forum was not the unanimous agreement of all parties, but the bringing together of individual divergent experiences to form a common understanding of student performance in a subject area that is truly larger, and broader, than its individual parts.

Initial Procedure

Following the approval of the standard-setting methodology, portfolios with adequate evidence were identified by the Office of Superintendent of Public Instruction Alternate Assessment Specialist. Forty scored portfolios with sufficient evidence had been copied for range finding activities for 2001 and 2002. These portfolios had also been used as training sets during training of portfolio scorers. The total scores on the portfolio content entries ranged from 5 to 20 points. OSPI staff recorded the scores for each of five scoring dimensions in each content area of the portfolios. Sixty-two scoring patterns were found. Frequencies of student score profiles were generated from this group. Fourteen portfolios were selected to be used for standard-setting activities; this subgroup represented 32 different scoring patterns. Standard-setting was conducted using only these portfolios with adequate evidence.

Panel Meeting: Step 1 - Training

The purpose of this step was to give information about what the task of standard setting entails and about how student results will be used in reporting and evaluation. Participants introduced themselves and provided some information about their backgrounds to help the panelists get to know each other and provide a perspective for the various backgrounds of the panelists. The process and criteria for selecting panelists was reviewed to explain why the panelists were there and provide insight into what factors are important to the standard-setting process. Panelists were told that their job was to recommend how to place student results in four levels of performance. Background on the ESEA and IDEA requirements was provided.

Contractor and OSPI staff described the standard-setting process, and focused on the general nature of standard setting. This helped the panelists understand the overall process and the iterative nature of the standard setting. This session did not focus on specific procedures that the panelists were to use later, but attempted to give them an appreciation for the group judgment process and the panelist’s role in the process.

Panel Meeting: Step 2 - Review of the Assessment Material

Panelists became familiar with the assessment at this point. Panelists were told that the portfolio is a collection of evidence of student work and that only the evidence that is present can be considered. The panelists reviewed the participation criteria for WAAS so that they were familiar with the type of student who would participate in the portfolio assessment (see Appendix A).

Panelists were introduced to the content validity evidence for the assessment and the scoring processes. Exemplars of student work for each score point were reviewed by the panelists in a group setting so questions about scoring and rubrics could be answered. A summary of the development process was provided. Panelists were informed as to the use of standards across assessments and issues of comparability. The Portfolio Scoring Summary (Appendix B) and the complete EALR Extensions were distributed to panelists (A summary overview of the approach to EALR Extensions in included in Appendix C, and the entire document is available upon request). The Scoring Summary dimensions, rubrics, and EALR Extensions were developed and refined by a special education stakeholder curriculum group during the piloting phase of portfolio development. Both of these documents were explained in detail to panelists.

In 2003, evidence for student performance in science will be collected for the first time. Given the nature of the standard-setting procedures and that the requirements for portfolio entries are similar for each content area, standards were set for all content areas including science. Panelist raised questions related to scoring and were reminded that the current task before the committee is to set standards and not to alter scoring procedures.

Panel Meeting: Step 3 - Understanding the Definitions of the Standards

This step was designed to introduce panelists to the definitions of the academic achievement standards. Panelists did a brainstorming exercise to help them think of students and student work that typify the definitions of each standard and the performance of students who are at the standard. Panelists did not write or re-write the definitions at any time. This step only served to familiarize panelists with definitions that have previously been determined, and to help the panelists think about students who are at each of the levels.

Academic Achievement Standard Descriptions were provided to the panelists that identify student performance in four levels (See Appendix D). These descriptions were written by the Alternate Assessment Specialist, using National Assessment of Educational Progress (NAEP) descriptors and the portfolio scoring criteria as a framework. The basis for these definitions was reviewed, and the panelists discussed the definitions until the levels or categories of student performance were clearly distinguished from each other and no ambiguity regarding their characteristics remained.

Panelists were assigned into three groups of five or six to discuss the descriptors. Each subgroup was composed of at least one person who had portfolio-scoring experience, at least one person on the Advisory Committee, and others who were new to the portfolio process. Panel subgroups discussed and proposed revisions to the descriptors. A record of these suggestions was made so that they could be discussed by the entire group and used as a reference during the standard-setting process. The three subgroups shared their revisions with the entire panel; the revised performance descriptors listed by each subgroup were very similar. All proposed revisions were posted for use in the standard setting process.

Panel Meeting: Step 4 - Determination of Achievement Levels Based on Scoring Guides and Scoring Patterns

Panelists received rating sheets and were instructed in the process of completing the sheets. Working with the table of frequencies of scoring patterns and with the existing descriptors for each scale, panelists decided on the category ("meets" or "not meets" proficiency) for each score pattern. The level placement for each pattern and the minimum total score in each category (level 2, 3, 4) were recorded. This set of ratings was placed under Round 1 Levels on the rating forms (See Appendix E).

Panel Meeting: Step 5 - Holistic Classification of a Range of Student Portfolios

Participants received feedback on the overall panel cut scores they had established. The cut scores for the panel were based on the mean of the minimum total scores for each proficiency category. Panelists were then given the set of 14 scored portfolios that were exemplars of the patterns already categorized. Working individually, they reviewed the portfolio evidence, total score, and scoring patterns. They made decisions for each portfolio and reset, if needed, the minimum score for each proficiency level. They then reconvened into subgroups (same members as those on the previous day) to discuss ratings and make final decisions.

The discussion at this point included whether or not to set a minimum score for the Part I score for each content area. This discussion related to concerns that Part I scores were used once for each content areas, but Part II scores were used across all content areas, with the possible effect of more heavily weighting of Part II scores.

The Washington Alternate Assessment Portfolio Scoring Summary (Appendix B) consists of two parts. Part I: progress on IEP Skills is scored separately for each content area entry. The score points are included below.

Part I: Progress on IEP Skills
(Progress on IEP skills scored separately for each content area entry.)

Part II: Student Generalization of Skills has four dimensions that are scored across the entire portfolio. The dimensions and score points are shown below.

Part II: Student Generalization of Skills
(These dimensions are scored across the entire portfolio.)

The discussion focused on the implication of Part I scores, which reflect on student progress on IEP skills linked to the EALRs, and if Part I scores had appropriate weighting in final determination of performance. Since the Part II dimensions are applied to all content areas, there was some concern that there was an effect of weighting on Part II scores that was not intended. The panel did not reach a conclusion at this time, but tabled the discussion for later in the process. Then, each panel member completed the rating form for Round 2.

Panel Meeting: Step 6 - Review of Impact Data

Panelists were presented impact data in the form of frequencies for each score pattern generated from 2002 portfolio scores, as well as statewide performance data to judge the impact of group standards, which included the entire population of students assessed that satisfied state-determined completion criteria. The panelists also received frequency distributions of total scores for the state and scoring patterns. Panelists discussed the impact of standards on the state.

After participants discussed the impact data they resumed the discussion of Part I and Part II score effects. After reviewing the Academic Achievement Standard Descriptions, the panel agreed on a method for determining standards, and discussed possible decision rules to allow for consideration of the Part I score in determining achievement levels. The decision rules as proposed by the Panel are provided in the Results section, and allow for a two part decision based on overall score balanced by the score on Part I.

Panelists were given their Final Rating Sheets and asked to make any changes they wish on the basis of the impact data and group discussions. Panelists were advised that this was the last round of adjustments. Panelists were allowed to change the raw score value of the group cut scores according to this new information. No group consensus was pursued. Participants turned in their final Round 3 cut score recommendations. The final cut scores were calculated as the mean of the minimum total scores for each proficiency category.

The academic achievement standard descriptors were the reference documents used during steps 4, 5, and 6 for checking to see if particular score patterns and cut scores indicated that the student had met the standard.

Panel Meeting: Step 7 - Evaluation of the Standard-setting Process

Panelists were given evaluation forms to complete and open-ended comments were encouraged. A summary of the responses is given in Table 2. The eight evaluation questions were rated on a 1 to 5 scale, with 1 indicating Not At All to 5 indicating Completely. All of the questions had a majority of responses of 4 or 5. Six panelists included additional comments. Four of the six comments were positive, such as great two days, great process, good interaction. One panelist made a suggestion for training the panelists on the general education curriculum and related (WASL) assessment procedures. One panelist felt that some bias was present (men seemed to be listened to more so than women.)

Based on the work of this panel, alternate achievement standards were set for the alternate assessment portfolio. Cut scores were set using the total score from adding the score for each part one score (Part I score - Progress on IEP Skills score for the content area) to the total score for the Part II dimensions (Modifications and Adaptations, Settings and Contexts, Social Relationships, and Self-Determination). Table 3 shows the mean score from the standard setting committee for Round 1 and Round 2.

The second part of the decision rule requires a minimum score on the first scoring dimension (Progress on IEP Skill). The achievement level for any subject cannot be more than 1 level higher than the subject dimension score. That is, a portfolio with a 1 in the subject dimension cannot be in an achievement category higher than 2. A portfolio with a score of 2 in the subject dimension cannot be in an achievement category higher than 3. Table 4 summarizes the decision rule. Portfolios with insufficient evidence are treated as zeros and students are reported as not meeting the standard for accountability purposes.

Appendix F shows the relationship among the various total scores, performance level, and pattern of various scores. Table 5 summarizes the actual 2002 WAAS portfolio results categorized in each of the four levels of performance using the decision rule in Table 4.