Source: The Center for Universal Design, North Carolina State University, 1997.

The application of philosophies from one field to another is never an exact transfer. Rather, general ideas are often manipulated by the receiving field in order to facilitate useful recommendations. Nevertheless, a clear connection exists between Universal Design of architecture and Universal Design of assessments. As stated above, the field of architecture had, before Universal Design, an egregious history of building a structure, then only later retrofitting it to be more inclusive when forced by law. The assessment community has followed suit in its practices, by often creating tests that need to be changed for students with diverse linguistic backgrounds or students with disabilities.

Changes made to tests, or accommodations, are extremely controversial. Testing accommodations are defined by Tindal and Fuchs (1999) as "changes in standardized assessment conditions introduced to level the playing field for students by removing the construct-irrelevant variance created by their disabilities. Valid accommodations produce scores for students with disabilities that measure the same attributes as standard assessments measured in non-disabled individuals." (p.8).

A theoretical assumption in the field is that accommodations will validate student test results. This assumption is hinged on the premise that the design of the test was such that a student’s disability or language ability would hinder her or his ability to demonstrate knowledge. This being the case, accommodations would then, as Tindal and Fuchs noted, "level the playing field." Bielinski, Thurlow, Ysseldyke, Friedebach, and Friedebach (2001) found, however, that the use of accommodations may alter a test to an extent that the accommodated and un-accommodated items are no longer comparable.

A challenging situation then arises about how to create equal opportunities for success on large-scale tests without changing constructs so much that student results are not comparable. Returning to architecture, Mace described similar issues when he first advocated the idea of Universal Design. Were Ron Mace a test designer, he may have suggested that tests be designed better from the start, rather than having to be retrofitted for individual users. Such was the philosophy adopted by NCEO when exploring Universal Design of assessments

Universal Design of Assessments

The perplexing question of how to make assessments more accessible to all students required a vast literature review that relied heavily on information from psychometric, special education and English language learner sources, but also looked outside of assessment-related literature for answers. Information from the fields of literacy, vision, and graphic design demonstrates that the way in which printed documents, instruments and materials are designed have great impact on how people perceive, interact, and perform on them.

Universally designed assessments, as described by Thompson, Johnstone, and Thurlow (2002) are "designed and developed from the beginning to allow participation of the widest possible range of students, and to result in valid inferences about performance for all students who participate in the assessment. Universally designed assessments add a dimension of fairness to the testing process" (p. 5). According to the Heubert (1999), such fairness and front-end design considerations must be continued throughout the assessment process from design to administration to scoring and interpretation.

The philosophical background for Universal Design of assessments is found in the Standards for Educational and Psychological Testing (AERA, APA, NCME, 1999), which state that "all examinees be given a comparable opportunity to demonstrate their standing on the construct(s) the test is intended to measure. Just treatment also includes such factors as appropriate testing conditions and equal opportunity to become familiar with the test format, practice materials, and so forth... Fairness also requires that all examinees be afforded appropriate testing conditions" (p. 74, cited in Thompson, Johnstone, & Thurlow, 2002).

Similar to the Center on Universal Design’s Principles, NCEO’s Thompson, Johnstone, and Thurlow (2002) developed a structure for guiding policy and practice of assessment called elements. The authors’ Elements of Universally Designed Assessments are compared to the Center on Universal Design’s (1997) Principles in Table 2.

Table 2. Relationship Between Principles of Universal Design and Elements of Universally Designed Assessments

Source: Thompson, Johnstone, and Thurlow (2002).

The elements are drawn from best practices that are accepted in the field of measurement but demonstrate a succinct collection of recommendations that relate to existing best practices in architecture. As noted by Thompson, Johnstone, and Thurlow (2002), the elements are likely to undergo changes as research increases the knowledge base of the field. Such changes are natural because, according to the Heubert (1999), "research and development in the field of educational testing is continually experimenting with new modes, formats, and technologies" (p. 202). A description of each element of Universally Designed assessments follows.

Element #1. Inclusive Assessment Population: NCEO’s first element replicates the inclusive nature of the Center for Universal Design’s principle of equitable use. In the case of large-scale assessments, current legislation dictates that all students need to be included in statewide accountability systems. Such inclusion ensures that students will have equal access to the public and private benefits conferred by schooling.

The American Educational Research Association, American Psychological Association, National Council on Measurement in Education (1999), and National Research Council (see Heubert, 1999) called for tests that are conceptualized for the contexts of the entire population to be assessed. In practice, this translates into piloting, field testing, and norming tests with the explicit goal of including students with a wide range of disabilities, students with limited English proficiency, and students across racial, ethnic, and socioeconomic lines. When doing so, constructs and standards should be maintained, but design features selected should reflect end users.

Element #2. Precisely Defined Constructs: Fidelity to constructs is important for two reasons. First, when test designers remain true to constructs, they are able to remove all non-construct oriented sensory, emotional, and physical barriers to students demonstrating competence in a particular area. By removing such barriers, states are better able to determine where needs for educational improvement lie. Constructs that are clearly defined are essential for making sound and valid educational decisions based on assessment results.

Examples to the contrary demonstrate the continued need for explicitly labeling constructs to be tested. In the area of mathematics, research has demonstrated that students are often limited in their ability to demonstrate mathematical knowledge because of the reading level of items. Calhoun, Fuchs, and Hamlett (2000), Harker and Feldt, (1993), Koretz (1997), and Tindal, Heath, Hollenbeck, Almond, and Harniss (1998) all found that students with reading difficulties scored higher on tested constructs in math when questions were read to them than when they read tests independently (all students in these studies had reading difficulties). Shorrocks-Taylor and Hargreaves (1999) suggested that the language used in questions on tests that assess subjects other than language become as "transparent" as possible to reduce construct irrelevant bias, or the "degree to which test scores are affected by processes that are extraneous to its intended construct" (AERA, APA, NCME, 1999).

Element #3. Accessible, Non-Biased Items: A biased item is one that is presented in a way that disadvantages particular test takers. Popham and Lindheim (1980) suggested that items can minimize unfair advantage to particular students by:

• Ensuring that required student responses are a valid demonstration that students have mastered content (curricular congruence).

• Ensuring the likelihood that students will have had the opportunity to learn the desired outcome of the item (instructional sensitivity).

• Ensuring that the socioeconomic status or inherited academic aptitudes the student possesses are not the dominant influence on how the student will respond to an item (out of school factors).

• Ensuring that the item’s content does not offend or unfairly penalize students because of personal characteristics such as race, gender, ethnicity, [disability,] or socioeconomic status (bias).

Bias in items can be examined both qualitatively and quantitatively. Geisinger and Carlson (1992), Popham (2001), and Popham and Lindheim (1980) all suggested that sensitivity review panels are utilized to determine whether particular items are biased against a particular group. Such panels consist of representatives from various ethnic and socioeconomic groups. In addition to such groups, sensitivity panels are strengthened by members that represent persons with disabilities and English language learners.

Bias can also be examined using post hoc analyses such as Differential Item Functioning (DIF) and Item Response Theory (IRT). Differential Item Functioning tests determine whether students with equal ability but representing different groups do not have the same probability of responding correctly to test items. DIF analysis has traditionally been used to detect the differential function of an item according to group identity (e.g., race, gender, disability). IRT is used to predict a student’s probability of answering an item correctly or incorrectly based on particular student characteristics.

Element #4. Amenable to Accommodations: Mace (1998) clarified that in Universal Design there still may be a need for post-design retrofitting to meet the needs of particular consumers. Likewise, while Universal Design of assessments may reduce the need for accommodations, it will never fully eliminate the need to make certain changes to the standard administration of tests. One example is the use of Braille tests. While Braille tests have obvious text differences, certain design features of the standard test improve the correspondence of Braille and standard tests. Features that improve correspondence include:

• Avoiding the use of construct irrelevant graphs or pictures.

• Avoiding the use of vertical or diagonal text.

• Not placing keys and legends at the left or bottom of the item where they are more difficult to locate in Braille formats.

• Avoiding items that depend on reading of graphic representations (such as blueprints, furniture in a room) that do not also have verbal/textual descriptions that can be translated into Braille.

• Removing items that include distracting or purely decorative pictures, which draw attention away from the item content.

Furthermore, a test is amenable to accommodations if it has "built in" accommodations for all students. For example, two common accommodations for students are extended time on tests and the use of manipulatives. If a test is not a speeded test, then time limits can be removed for all students. Second, if the construct of the test does not call for manipulative-free computation, then all students can be given the opportunity to use manipulatives. The above-mentioned examples demonstrate how different elements of Universal Design at times overlap to create more accessible testing for all.

Element #5. Simple, Clear, and Intuitive Instructions and Procedures: Tests that attempt to gauge student knowledge or performance on a particular task are invalidated when students are unclear of the tasks they must perform to demonstrate knowledge (ADDA, 2001; Elliott, 1999; Willingham, et al., 1988). Clarity of instructions is improved when sample items, examples, and criteria for scoring are made clear to students from the very beginning (AERA, APA, NCME, 1999; Grise, Beattie, & Algozzine, 1982). Tindal and Fuchs (1999) suggested that test designers should question whether it will "be possible for all students to work independently throughout the test?" And "are directions easy to follow?" as preliminary guides for determining whether instructions are simple, clear, and intuitive.

Element #6. Maximum Readability and Comprehensibility: Readability is a much-disputed term because formulae are often calculated to quantify the readability of a text or passage without considering the qualitative elements of that passage. Rakow and Gee (1987) treaded carefully on the description of readability, calling it an "estimate of probability of comprehension by a particular group" (p. 28). According to such estimates, readability generally increases when sentence lengths are reduced, words per line are minimized, and multi-syllabic words are used sparingly. While such estimates are generally reliable for texts and passages, they are less so for test items with only one or two sentences. In the case of items, Popham and Lindheim (1980) and Rakow and Gee (1987) suggested concentrating on features of text such as logical organization of ideas and clarity of message. These features are difficult to quantify, but Rakow and Gee (1987) suggested determining students’ prior experiences, achievement, and interests as well as text features when determining readability of a passage.

Gaster and Clark (1995) recommended eight considerations when attempting to improve readability. These were:

• Use simple, clear, commonly used words, eliminating any unnecessary words.

• When technical terms must be used, they should be clearly defined.

• Compound complex sentences should be broken down into several short sentences, stating the most important ideas first.

• Introduce one idea, fact, or process at a time; then develop the ideas logically.

• All noun-pronoun relationships should be made clear.

• When time and setting are important to the sentence, place them at the beginning of the sentence.

• When presenting instructions, sequence steps in the exact order of occurrence.

• If processes are being described, they should be simply illustrated, labeled, and placed close to the text they support.

Many of Gaster and Clark’s recommendations mirror the philosophies of the recent "Plain language Movement." Plain Language approaches have been applied to testing situations by Brown (1999) in an effort to reduce construct irrelevant variance based on student reading ability. Brown suggested that editors: reduce excessive length, eliminate unusual or low frequency words, avoid ambiguous words, avoid irregularly spelled words, avoid unclear signals about how to direct attention, and mark all questions clearly.

Element #7. Maximum Legibility: Legibility refers to the capacity with which items are able to be deciphered with ease. In terms of testing, three general categories of legibility are necessary to lead to "maximum" legibility. The first category is the legibility of text. Table 3 demonstrates characteristics of legible text culled from research on vision, graphic design, and literacy. Each dimension of legible text (contrast, type size, spacing, leading, type face, justification, line length, and blank space) is defined and examples are provided to illustrate characteristics.

Source: Thompson, Johnstone, and Thurlow, 2002. 

The second category of legible text refers to the comprehensibility of graphs, tables, and illustrations. Best practices in graphic design guide suggestions for improving the legibility of visual materials on tests. For example, Gregory and Poulton (1970) suggested information is more quickly found on graphs when plot lines and axes are clearly labeled. Shriver (1997) concurs, noting that clearly-designed quantitative graphic information creates a context for appropriate interpretation of data.

Criteria of utility and design lead to the appropriate use of illustrations in text. Sharrocks-Taylor and Hargreaves (1999) found that there are three different types of illustrations found in tests:

• Decorative illustrations that are not related to the questions and serve no instructional purpose.

• Related illustrations that have the same context as the questions and are used to support text and emphasize ideas.

• Essential illustrations that are not repeated in the text, but the text refers to them, and they have to be read, or worked with, to answer the question.

Because certain students are easily distracted or may be led astray by decorative illustrations, such illustrations may create barriers to success and should therefore be removed from tests. Silver (1994) and West (1997) recommended that when related and essential illustrations are used, they be placed directly next to the question to which they refer. Similar principles apply to computer-generated images. Szabo and Kanuka (1998) suggested that computer graphics should comply with principles of unity, focal points, and balance in order to increase completion rates on tests.

Finally, even when useful to the item and well-designed, illustrations are most effective when they match the cultural schema of the end user. Not only should graphics be culturally appropriate in terms of experience (Shriver, 1997), they should also be communicated respectfully, considering the student’s cultural norms and belief systems (Schiffman, 1995).

The final category of legibility is the legibility of response formats. Response formats are the portions of tests where students are asked to respond to items. In many norm-referenced tests, students are asked to respond on computer-ready bubble sheets. Such formats are inherently biased against students with fine motor difficulties or low vision. Thus, marking answers directly on the page is recommended for both people with low vision and Braille users. Larger circles on which to "bubble" are recommended for all users. As Willingham et al. (1988) noted, many students with learning disabilities have a lack of body awareness and poor directionality. Larger bubbles (or answering directly on test) could help to mitigate some of these issues.

Research in the area of response formats continues today, but several trends have emerged over the past 20 years. Grise et al. (1982) found that flattened, horizontal bubbles were more useful for students with learning disabilities than circular bubbles. Rogers (1983) and Tindal et al. (1998) both found that separate answer sheets were problematic for students in grades 1-3. Results from grades 4-6, however, were mixed. Tindal et al. (1998) found no difference in scores for grade 4 students using separate answer sheets. Rogers (1983) determined that grades 4 and 5 students could use separate answer sheets effectively, provided they were given special instructions. Veit and Scruggs (1986), however, found that fourth grade students with learning disabilities took significantly more time to complete tests when bubbling was required. Muller, Calhoun, and Orling (1972) determined that students in grades 3-6 made fewer errors when allowed to answer directly on answer sheets. Overall, separate answer sheets are a practice that speed scoring time and decrease costs of large-scale tests. There is no research, however, demonstrating that separate answer sheets are superior to answering questions directly on the test.

When the legibility of text, graphic information, and response formats are improved, the overall legibility of the test increases. To do this, format and font as well as graphics and illustrations deserve close attention. Furthermore, the way students are asked to respond must not be taken for granted. The combination of all elements provides a framework for examining tests and their level of accessibility. Individual items may not have direct links to all elements, but Universal Design of assessments is presented here as a general framework for improving the design (and thereby accessibility, comprehensibility, and validity) of tests.

Although Universal Design of assessments is a relatively recent addition to assessment literature, policymakers at the state and national level have begun to mention the philosophy of Universal Design or its elements. References to Universal Design in testing can be found in the State of California’s educational policy, which requires educators to employ strategies of "Universal Design" of education in both classrooms and assessments (State of California, 2003) and the State of Vermont’s educational assessment Request for Proposal (RFP), which calls for "Universally Designed Assessments" using Thompson et al.’s Elements of Universally Designed Assessments (State of Vermont, 2003).

Every new idea that will be employed in policy and practice carries a burden of proof. Specific and scientific research must be documented on all positions proposed in the field of education. A recent study, funded by the United States Department of Education, Office of Special Education Programs aimed to determine the efficacy of Universal Design. Although Universal Design literature and funding have a specific leaning toward disability issues, Thompson, Johnstone, and Thurlow (2002) stated that Universal Design could improve assessment for every student. Such a proposition was put to the test in United States Department of Education Project
# H324B020025, which was a student-initiated award to conduct research on this issue.

Research Methodology

A mixed-method design was chosen to test hypotheses and gain a greater understanding of test-related issues for students. The methods, which combine experimental and descriptive techniques, asked the research questions:

The purpose of the project was to determine whether eliminating construct irrelevant material in tests would provide a more accurate measure of student performance. By eliminating superfluous, non-construct relevant information and requirements, a universally designed test was examined to determine whether it was more accessible to students with disabilities and provided a clearer assessment picture of all students, including those who do not qualify for special accommodations (students with mild disabilities, low performing regular education students who have "fallen through the [service] cracks" and students with limited English proficiency).

Typically, test scores for students with low socioeconomic status, English language learners, and students in special education programs have been lower than national averages. Factors such as poor teaching, poverty (Willems, 1986), and limited English proficiency have all been suggested as reasons for poor performance. While personal reasons for student failures are often investigated, the design of tests has rarely been scrutinized as a reason for student under-performance.

To address the research questions, the proposed research project was divided into two studies. Study One was a comparison study of results from a cohort of students on two tests. The first (control) test was a practice mathematics test comprised of actual (released) statewide assessment items. The second (experimental) test was a revision of the same test, adjusted to incorporate Universal Design principles. This test was created by a team of experts. Four community members served on this team. Three of the participants were parents of students in special education programming in the school district. The fourth was a teacher in the district who has dyslexia. Each was considered to have unique perspectives that were solicited for the project. Each parent represented one of the major ethnic groups of the area.

All experimental items were generated from the constructs of the control test items. While constructs remained constant in the experimental test, design and administration features were changed in accordance with Universal Design principles. Students with disabilities were examined on both tests according to their IEP guidelines (i.e., with the accommodations indicated in their IEPs).

The second study was a qualitative analysis of the difference in student experiences in the two versions of the test. Students included in the second study were those who demonstrated a 1.5 standard deviation improvement from control to experimental tests. These students were interviewed about their perceptions of the two tests.

As stated above, the purpose of this study was to determine whether the principles of universally designed assessment would more validly assess a sample of all students, and to specifically examine students who were economically poor, mostly English language learners, ethnic minorities, and students with disabilities. The sample was taken from four schools in the U.S. Southwest. Two of the schools were in a small town of approximately 20,000 and two schools were in rural areas. All of the schools were adjacent to Native American Tribal Land.

In total, 231 students participated in the study. Demographic data were missing for 18 subjects. Of the data available, 165 were Native American, and 25 were Latino/a, and 23 were Anglo/a/white. Thirty-one students had specific learning disabilities, 109 were English language learners, and 132 were reading below grade level (according to local assessments).

Study 1 was divided into two parts, each part using counterbalancing methodology to ensure valid experimentation. Each student in the sample completed two tests, one traditionally designed and one universally designed. Half of the cohort took the traditionally designed test (made up of released statewide assessment items) according to standard testing procedures. Students with IEP accommodations took the test with all specified accommodations. Later, this group took a second test redesigned using Universal Design principles. The other half of the cohort took the tests in reverse (universally designed first and traditionally designed second) to prevent a "practice effect" (Edelman, 1996) and inflation of scores. To maintain inter-test reliability, all items were drawn from other (released) statewide test items or created by a team of experts using the process described below. Table 4 demonstrates the testing schedule.

Table 4. Testing Schedule

The creation of the universally designed test followed a systematic procedure that was created for this research project. Such a procedure may be useful to test-designers when attempting to include Universal Design Elements into assessments. The step-wise procedure was performed as follows:

1. Items were selected for revision from released statewide tests.

2. The team of experts (referred to as the "Advisory Board") was provided information and a training session on the Principles of Universal Design. The Advisory Board then examined the items for perceived design flaws. Design problems and suggestions for improvement were re-submitted to the researcher. The Advisory Board was asked to make changes based on their knowledge of Universal Design and their personal insights based on their child’s disability, their own disability, or their cultural background.

3. A mathematics expert at a large research university examined all items on the "traditionally designed" test and marked what he perceived the construct to be measured.

4. Experts on Universal Deign in testing from NCEO examined items for design flaws and possible improvements.

5. Test items were re-designed, keeping constructs constant but removing construct-irrelevant information, changing font size, paper color, and re-arranging numeric combinations (e.g., an item that called for students to add 4,533 + 3,205 might be changed to 3,054 + 4,715).

An example of changes made to a particular item is found in Table 5. Note that some changes were superficial (such as font size) while others greatly changed the administration of the test (response formats and timing). None of the changes disrupted constructs. Original tests were not speeded but timed for administration ease.

Table 5. Item Re-design Process

Tests were administered to students in using the methods described above. Student scores were then analyzed to determine whether: (1) student performance was affected by test design (claims of causation are possible because experimental and control groups were present), and (2) students were affected differently by ethnicity, disability status, English language learner status, and reading ability.

Group means were calculated for both the traditionally-designed test and the universally designed test. Because the sample took both tests on the same day (to prevent sample deviation or unequal opportunities for exposure to test material), a matched sample t-test methodology was chosen to determine whether there was a statistically significant difference between group means. T-test results are reported in the Findings section. To boost the explanatory power of statistical results, a Cohen’s d effect size was also calculated, with results reported in the Findings section.

Although the sample was purposively chosen to reflect populations with high numbers of ethnic minorities, English language learner students, and students with disabilities (all who traditionally under-perform on large-scale tests), samples within schools themselves were random. Because of this, student sub-populations were unequal, making between-group statistical inferences impossible. Data were disaggregated and descriptive results for various sub-groups are presented in the Findings section of this paper.

Study 2 was a qualitative study, intended to give voice to students (Fetterman, 1998) concerning the traditionally and universally designed tests. The methodological intention of this study was to analyze student perceptions about traditionally and universally designed tests. Quantitative data were valuable in determining the effectiveness of different tests, but qualitative data provided perspectives as to why students are performing in particular ways. A series of interviews was conducted with students with and without disabilities in order to gather their perceptions on the two tests.

Following the second test, control and experimental data were compared. A group of 23 students was chosen and individually interviewed by the Student Researcher. Students who demonstrated a 1.5 standard deviation change in score between control and experimental tests were selected as the purposive sample for this study (Patton, 1990).

Interview questions addressed three major themes: comprehension of test material, test design, and overall large-scale testing issues. For the first and second themes, comprehension of test material and design features, students were asked a series of questions about the two tests they took. Students were provided actual protocols to review and were asked to share their perceptions of the two tests.

Questions on the third theme asked students to reflect on their experiences with large-scale tests and to describe their feelings related to such experiences. Building on these feelings, students were asked to provide advice to test makers for ways to improve testing to make it more child-friendly. A sample of all interview questions is found in Appendix A. All interviews were conducted in a structured interview format (Bogdan & Biklen, 1992) to ensure that prompts were standardized for each student.

All interviews were transcribed and read for content. Qualitative themes arose from data. Themes (and quotes to exemplify them) are provided in the Findings section. Qualitative research, as a methodology, cannot provide data to make causal claims. Rather, in this study, qualitative themes are used to illustrate statistical information and to help the reader better understand the end users of tests whose perspectives may be different from those of educators, policymakers, or assessment designers.

Findings

Results of this research demonstrated that applying the elements of Universal Design of assessments does affect student performance on tests. Overwhelmingly, this effect is an increase in achievement. Such a finding proves that, if constructs are held constant, the design of a test can influence how a student performs. Implications of such a finding are discussed in detail in the final section of this report, and raise important issues in the current era of high-stakes testing.

Upon completion of tests, all student scores were tabulated and entered into SPSS software. A simple sign test demonstrated that 155 students of the 231 sampled scored higher scores on the universally designed test. Conversely, 51 students had lower scores on the universally designed tests, and 25 students had the same score on the traditionally and universally designed tests. Student demographics appeared to follow general sample demographics for both students who did and did not show "improvements" from the traditional to universally designed assessments. Of the 155 students with improved scores, all were statistically significant, yet only 17 students had statistically significant negative scores (this is most likely an effect of the relatively small number of decreased scores).

Next, mean scores were tabulated for subgroups. Target subgroups were those mentioned in No Child Left Behind guidelines and those of particular interest to the researcher. Subgroup means that were tabulated were: Navajo students; Latino/a students; White, non-Hispanic students; students with disabilities; non-disabled students; English language learner students; English language proficient students; students who read at or above grade level; students who read below grade level; and students who are both English language learners and have disabilities. Because students often belonged to more than one subgroup, statistical inferences were impossible. Every subgroup demonstrated improved performance on the Universally Designed test. Mean scores for both tests as well as standard deviations are presented below.

Table 6. Sub-group Results on Traditionally and Universally Designed Tests