Access to Computer-Based Testing for Students with Disabilities
NCEO Synthesis Report 45
Published by the National Center on Educational Outcomes
Prepared by:
Sandra J. Thompson • Martha L. Thurlow • Rachel F. Quenemoen • Camilla A. Lehr
June 2002
Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:
Thompson, S. J., Thurlow, M. L., Quenemoen, R. F., & Lehr, C. A. (2002). Access to computer-based testing for students with disabilities (Synthesis Report 45). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Synthesis45.html
Executive Summary
Called the “next frontier in testing,”
computer-based testing is being promoted as the solution to many of states’
testing problems. With pressure to find more cost effective and less labor
intensive approaches to testing, states are seeing computer-based testing as a
way to address the increasingly challenging prospect of assessing all students
in a state at nearly all grades. Computer-based testing is viewed with optimism
as an approach that will make testing less expensive in the long run, and that
will produce better assessments of the wide range of students who must now be
included in state and district assessments.
Unfortunately, most states and testing
companies have not specifically considered the needs of students with
disabilities as they pursue computer-based testing. Often, the approach has
simply been to take the paper and pencil test and put it onto a computer. This
is not enough. Poor design elements on the paper test will transfer to the
screen, and there will be additional challenges created by the move as well,
challenges that may reduce the validity of the assessment results and possibly
exclude some groups from participation in the assessment.
This paper recognizes both the opportunities
created by the new frontier of computer-based testing, but also identifies the
challenges. Research findings and accommodations considerations are also
addressed, with the end result being a process and considerations for the
initial transformation of paper/pencil assessments to inclusive computer-based
testing.
The recommended process for a good
transformation of a paper and pencil test to computer-based testing assumes
first that the principles of universally designed assessments have been
followed. Then, the five step that are recommended (and discussed in the paper)
are:
Step 1. Assemble a group of experts to guide
the transformation.
Step 2. Decide how each accommodation will be
incorporated into the computer-based test.
Step 3. Consider each accommodation or
assessment feature in light of the constructs being tested.
Step 4. Consider the feasibility of
incorporating the accommodation into the computer-based test.
Step 5. Consider training implications for staff and students.
The paper also presents initial
considerations for common accommodations within the categories of
timing/scheduling, presentation, response, and setting.
On January 8, 2002, President Bush signed the
reauthorization of the Elementary and Secondary Education Act into law as the
“No Child Left Behind Act of 2001.” This Act requires states to have annual
assessments in place in reading and mathematics for all students in grades three
through eight by the end of the 2005-2006 school year, with science assessments
added by the beginning of the 2007-2008 school year. Only nine states currently
administer standards-based tests in both subjects across grades three through
eight (Quality Counts, 2002), setting an unprecedented opportunity for states to
enhance the participation of all students as they build and improve their
assessment systems. Increased requirements within the law for itemized score
analyses, disaggregation within each school and district by gender, racial and
ethnic group, migrant status, English proficiency, disability, and income will
challenge states to create new and more efficient ways to administer, score, and
report assessment results.
Computer-based testing has been called the
“next frontier in testing” as educators, testing companies, and state
departments quickly work to transform paper/pencil tests into technology-based
formats (Trotter, 2001). These efforts have occurred in a variety of ways and
for a variety of tests. For example, some educators have transferred all of
their classroom quizzes and tests into a computer-based format. The paper/pencil
version of the Graduate Record Exam™
has been replaced with a computerized version that is administered across a
variety of locations. NCS Pearson™
has developed eMeasurement™ Services—a
suite of tools that delivers tests and their results electronically.1 As a result of these advances, states are facing
pressure to create computer-based large-scale assessments (Russell, 2002). Some
states are investigating the possibility of computerized adaptive testing for
their statewide assessments, where the difficulty level of questions are
presented and adjusted based on whether students’ responses are correct.
According to Bennett (1998), “Whereas there is certainly a concerted move toward
technology-based large-scale tests, there is no question that this assessment
mode is still in its infancy. Like many innovations in their early stages,
today’s computerized tests automate an existing process without
reconceptualizing it to realize the dramatic improvements that the innovation
could allow. Thus, these tests are substantively the same as those administered
on paper” (p. 3).
With the dramatic increase in the use of the
Internet over the past few years, and with it, the considerable potential of
online learning (Kerrey & Isakson, 2002), assessment will need to undergo a
complete transformation to keep pace. According to the Web-based Education
Commission, “Perhaps the greatest barrier to innovative teaching is assessment
that measures yesterday’s learning goals…Too often today’s tests measure
yesterday’s skills with yesterday’s testing technologies—paper and pencil” (p.
3).
Experts suggest that the Internet will be
used to develop tests and present items through dynamic and interactive stimuli
such as audio, video, and animation (Lewis, 2001). Given this momentum, it is
not surprising that there is a trend toward investigating and incorporating the
Internet as the testing medium for statewide assessments. Bennett (2001) stated,
“The trend is clear: the infrastructure is quickly falling into place for
Internet delivery of assessment to schools, perhaps first in survey programs
like NAEP (National Assessment of Educational Progress) that require only a
small participant sample from each school, but eventually for inclusive
assessments delivered directly to the desktop” (p. 10).
As the trend toward computer-based testing
moves forward, it is important to focus carefully on the requirements of the
newly enacted No Child Left Behind Act of 2001, and on the assessment
participation requirements in the 1997 reauthorization of the Individuals with
Disabilities Education Act. In addition, a 1996 Department of Justice Policy
Ruling states that Titles II and III of the Americans with Disabilities Act
requires State and local governments to provide effective communication whenever
they communicate through the Internet. The Office for Civil Rights discussed the
provision of effective communication:
The issue is not whether the student with the disability is merely provided access, but the issue is rather the extent to which the communication is actually as effective as that provided to others. Title II [of the Americans with Disabilities Act of 1990] also strongly affirms the important role that computer technology is expected to play as an auxiliary aid by which communication is made effective for persons with disabilities (Pages 1-2, 1996 Letter; 28 C.F.R. 35.160 (a)).
In further clarification, the Office for
Civil Rights lists three basic components of effective communication:
“timeliness of delivery, accuracy of the translation, and provision in a manner
and medium appropriate to the significance of the message and the abilities of
the individual with the disability” (Page 1, 1997 Letter). This clarification
presents a significant and timely responsibility in the design of computer-based
testing.
For the full benefits of computer-based
testing to be realized, a thoughtful and systematic process to examine the
transfer of existing paper/pencil assessments must occur. It is not enough to
simply transfer test items from paper to screen. Not only will poor design
elements on the paper test transfer to the screen, additional challenges may
result that reduce the validity of the assessment results and possibly exclude
some groups of students from assessment participation.
This paper presents factors to consider in
the design of computer-based testing for all students, including students with
disabilities and students with limited English proficiency. We begin with the
opportunities and challenges presented by this “new frontier” in testing, and
then explore research about effective universally designed assessments and
technology-based accommodations, and relate this knowledge to computer-based
testing design features. Finally, we present a process and consideration for the
initial transformation of paper/pencil assessments to inclusive computer-based
testing.
Opportunities
Several advocates have articulated the
positive merits of computer-based testing. Some of the advantages over
paper/pencil tests that have been cited include: efficient administration,
preferred by students, self-selection options for students, improved writing
performance, built-in accommodations, immediate results, efficient item
development, increased authenticity, and the potential to shift focus from
assessment to instruction. This section describes each of these prospective
opportunities.
Efficient Administration
Computer-based tests can be administered to
individuals or small groups of students in classrooms or computer labs,
eliminating timing issues caused by the need to administer paper/pencil tests in
large groups in single sittings. Different students can take different tests
simultaneously in the same room.
Preferred by
Students
In an evaluation of testing experience,
students overwhelmingly preferred computerized testing to paper/pencil testing
(Brown & Augustine, 2001). Most students, regardless of group or ability,
believed that the computer was easier, faster, and more fun. Students also
responded that using a computer helped concentration by presenting only one
question at a time. A recent survey on computer use by students with
disabilities in Germany (Ommerborn & Schuemer, 2001) found several more
advantages than disadvantages to computer use.
Brown-Chidsey and Boscardin (1999)
interviewed students with learning disabilities and found that the computer
helped them deal with limitations that often interfered with the completion of
their work. The researchers concluded, “Students’ beliefs about computers are
likely to shape the extent to which instructional technology enhances their
achievement” (Brown-Chidsey, Boscardin, & Sireci, 1999, p. 4). A study at the
Boston College Center for the Study of Testing, Evaluation, and Assessment
(Trotter, 2001) found, “Students who are accustomed to writing on computers tend
to do better on computerized tests than on paper exams. Conversely, students who
don’t use computers often to write tend to do better when they complete their
tests on paper” (p. 3).
Self-Selection
Options for Students
Students have the option to choose features
on computer-based tests, including format features and built-in accommodations.
For example, Calhoon et al. found that “teachers are unlikely to provide a
reader to meet student needs because teachers prefer test accommodations that
require little individualization and do not require curricular or environmental
modifications” (p. 272). Other recent work on accommodations for English
Language Learners (Anderson, Liu, Swierzbin, Thurlow, & Bielinski, 2000; Liu,
Anderson, Swierzbin, & Thurlow, 1999) has shown that students may not want to
use certain accommodations (e.g., headphones to have instructions read in
English, bilingual dictionaries) unless they are provided in specific ways.
Teachers have reported that students with learning disabilities may opt not to
use certain accommodations at certain times because they are not seen as
helpful. Having the ability to self-select a technology-based reader or other
tool may provide students access to a necessary accommodation that may not be
offered currently, due to issues of convenience.
Improved Writing
Performance
As computers become more common in schools,
many of today’s students are accustomed to using computers in their daily work.
Students write and calculate on computers as easily and with more speed and
efficiency than previous generations could on paper. Research has shown that
writing on computers leads students to write more and revise more than writing
with paper/pencil (Daiute, 1985; Morocco & Neuman, 1986). Paper/pencil tests
that require writing may underestimate the writing ability of students who have
grown accustomed to writing on computers (Russell & Haney, 1997). In a survey of
computer use by students with disabilities in Germany, Ommerborn and Schuemer
(2001) found that the greatest advantage to students was the ease in which
computers allowed them to write essays. Several of the students surveyed said
that it was very difficult for them to write by hand.
Built-in
Accommodations
Computer technology has been touted as a tool
that can be used to empower students with disabilities (Goldberg & O’Neill,
2000). Specifically, computer-based testing has been viewed as a vehicle to
increase the participation of students with disabilities in assessment programs.
For example, the windows operating system supports a great variety of adaptive
devices (e.g., screen readers, Braille displays, screen magnification,
self-voicing Web browsers). According to Greenwood and Rieth (1994), the primary
strength of computer-based testing is its “potential for removing traditional
barriers to the inclusion of persons with disabilities in the assessment process
through adaptations and accommodations as well as through new forms” (p. 110).
Computer-based testing can provide
flexibility in administration for students with various learning styles. For
example, the National Research Council (NRC, 2001) found computer-based testing
to be effective for students who perform better visually than with text, are not
native English speakers, or are insecure about their capabilities. According to
NRC, “Technology is already being used to assess students with physical
disabilities and other learners whose special needs preclude representative
performance using traditional media for measurement” (p. 286).
Standardization of accommodated assessment
administrations can be facilitated by computer-based testing. According to
Brown-Chidsey and Boscardin (1999), “Using a computer to present a test orally
controls for standardization of administration and allows each student to
complete the assessment at his/her own pace” (p. 2). Brown and Augustine (2001)
cited educator appreciation of a computer’s ability to present items over and
over, in both written and verbal form, without the need for a non-standard (and
sometimes impatient) human reader. Several studies have shown the positive
effects of providing a reader for math tests (see Calhoon, Fuchs & Hamlett,
2000; Fuchs, Fuchs, Eaton, Hamlett, & Karns, 2000; Tindal, Heath, Hollenbeck,
Almond, & Harniss, 1998).
With the use of audio and video built into
computer-based tests, specialized testing equipment such as audiocassette
recorders and VCRs could become obsolete (Bennett, Goodman, Hessinger, Ligget,
Marshall, Kahn, & Zack, 1999). According to Bennett (1995), “Test directions and
help functions would be redundantly encoded as text, audio, video, and Braille,
with the choice of representation left to the examinee. The digital audio would
allow for spoken directions, whereas the video could present instruction in sign
language or speech-readable form. Among other things, these standardized
presentations should reduce the noncomparability associated with the uneven
quality of human readers and sign-language interpreters” (p. 10).
Finally, just as the use of accommodations on
paper/pencil tests has increased awareness and use of accommodations in the
classroom, so can opportunities to use the built-in accommodation features of
computer-based tests encourage and increase the use of those features in
classroom and other environments. For example, Williams (2002) believes, “It is
possible that new developments in speech recognition technology could increase
opportunities for individual reading practice with feedback, as well as
collecting assessment data to inform instructional decision making” (p. 41). In
addition, most computer-based tests have built-in tutorials and practice tests.
These tutorials provide students with both opportunities for familiarizing
themselves with the software and immediate feedback (Association of Test
Producers, 2000).
Immediate Results
One of the major drawbacks of state testing
on paper has been the long wait for results because of the need to distribute,
collect, and then scan test booklets/answer forms and hand score open-response
items and essays. Students tested in the spring often do not receive their
results until fall—nor do their teachers or schools. The results of
computer-based tests can be available immediately, providing schools with
diagnostic tools to use for improved instruction, and states with information to
guide policy. Even open-ended items can be scored automatically, greatly
reducing cost and scoring time (Thompson, 1999). According to a report by the
National Governors Association (2002), cost savings can result from “the
elimination of printing and shipping activities when paper testing ceases” (p.
7).
Efficient Item
Development
As computer-based testing becomes more
developed, item development will be more efficient, higher quality, and less
expensive (National Governors Association, 2002). Bennett (1998) believes that
at some point items might be generated electronically, with items matched to
particular specifications at the moment of administration. “Test design will
also be the focal point for responding to diversity. The effects of different
test designs on minority group members, females, …will be routinely simulated in
deciding what skills and which task formats to use in large-scale assessments”
(Bennett, 1998, p. 9). According to Russell (2002), “already, some testing
programs are experimenting with ways to generate large banks of test items via
computer algorithms with the hope of saving the time and money currently
required to produce test items manually” (p. 65). Baker (2002) cited several
research efforts that have significantly advanced the progress of schema or
template-based, multiple-choice development and test management systems (see
Bejar, 1995; Bennett, 2002; Chung, Baker, & Cheak, 2001; Chung Klein, Herl &
Bewley, 2001; Gitomer, Steinbert, & Mislevy, 1995; Mislevy, Steinbert, & Almond,
1999).
Increased
Authenticity
Computers allow for increased use of
“authentic assessments”—responses can be open-ended rather than just relying on
multiple choice. According to Bennett (1998), the next generation of
computer-based tests will be “qualitatively different from those of the first
generation. This difference will be evident in the test questions (and, in some
cases, the characteristics they measure), as well as in development, scoring,
and administrative processes” (p. 4, see Table 1). Bennett notes that many
Americans are now receiving their news from TV and the World Wide Web, with the
expectation that students will increasingly be able to process information from
a variety of sources, not just from print. Bennett also suggests that response
formats will shift dramatically, perhaps including problems in which a student
is not expected to find the best answer, but a reasonable one within certain
constraints.
Generation |
Key Characteristics |
| First-Generation Computer-Based
Tests (Infrastructure Building) |
Primarily serve institutional
needs Measure traditional skills and
use test designs and item formats closely resembling paper-based tests Take limited advantage
of technology |
| Next-Generation Computer-Based
Tests (Qualitative Change) |
Primarily serve institutional
needs Use new item formats (including
multimedia and constructed response), automatic item generation, automatic
scoring, and electronic networks to make performance assessment an integral
program component; measure some new constructs Allow customers to
interact with testing companies entirely electronically |
Generation “R” Test (Reinvention) |
Serve both institutional and
individual purposes
Integrated with instruction via electronic tools so that performance is
sampled repeatedly over time; designed according to cognitive principles Use complex simulations,
including virtual reality, that model real environments and allow more
natural interaction with computers |
Adapted from: Bennett, R.E. (1998). Reinventing assessment: Speculations on the future of large-scale educational testing. Princeton, NJ: Policy Information Center, Educational Testing Service.
Shifts Focus from
Assessment to Instruction
Bennett (1998) believes that eventually
large-scale assessment will join with instruction. “Decisions like certification
of course mastery, graduation eligibility, and school effectiveness will no
longer be based largely on one examination given at a single time but will also
incorporate information from a series of measurements” (p. 11). “By virtue of
moving assessment into the curriculum, the locus of the debate over performance
differences must logically shift from the
accuracy of assessment to the adequacy of instruction” (p. 12). Bennett
continues this line of thought in a 2001 article, “When well-constructed tests
closely reflect the curriculum, group differences should become more an issue of
instructional inadequacy than test inaccuracy. As attention shifts to the
adequacy of instruction, the ability to derive meaningful information from test
performance becomes more critical” (p. 2).
Challenges
Despite the potential advantages offered by
computer-based testing, there remain several challenges, especially in the
transition from paper/pencil assessments. First of all, the use of technology
cannot take the place of content mastery. No matter how well a test is designed,
or what media are used for administration, students who have not had an
opportunity to learn the material tested will perform poorly. Students need
access to the information tested in order to have a fair chance at performing
well. Hollenbeck, Tindal, Harniss, and Almond (1999) strongly caution that the
use of a computer, in and of itself, does not improve the overall quality of
student writing. They, and other researchers, continue to find significantly
lower mean test scores for students with disabilities than for their peers
without disabilities. Other challenges that must be overcome in order for
computer-based testing to be effective include: issues of equity and skill in
computer use, added challenges for some students, technological challenges,
security of online data, lack of expertise in designing accessible Web pages,
and prohibitive development cost.
Issues of Equity
and Skill in Computer Use
Concerns continue to exist in the area of
equity, where questions are asked about whether the required use of computers
for important tests puts some students at a disadvantage because of lack of
access, use, or familiarity (Trotter, 2001). Concerns include unfamiliarity with
answering standardized test questions on a computer screen, using buttons to
search for specific items, and indecision about whether to use traditional tools
(e.g., hand held calculator) vs. computer-based tools. According to Wissick and
Gardner (2000), “Students will not take advantage of help options or use
navigation guides if they require more personal processing energy than they can
evoke” (p. 38).
A survey on computer use by students with
disabilities in Germany (Ommerbon & Schuemer, 2001) found the cost of acquiring
and using a computer as the greatest barrier, with the second being a lack of
training opportunities. Students who needed assistive technology cited high cost
and lack of information as barriers to increased computer use.
The gap in access to technology—sometimes
referred to as the “Digital Divide”—is continuing to grow. According to Bolt and
Crawford, authors of Digital Divide
(2000, p. 98):
While over 80 percent of families with incomes of $100,000 or more have computers at home, only about 25 percent of those households with annual incomes under $30,000 have home access to computers. Demographically, this means that the digital revolution is in full swing in America’s wealthy suburbs and affluent sections of cities and towns, while in some of our poorest areas, it is a phenomenon that is at best heard about on television. The gap has widened considerably for computer ownership among racial minorities when compared with European-Americans. In the context of the overall racial digital divide, a low-income European-American child is three times more likely to have internet access than his or her African-American counterpart, and four times as likely as a Latino family in the same socioeconomic category.
Added Challenges
for Some Students
Some research questions whether the medium of
test presentation affects the comparability of the tasks students are being
asked to complete. Here are some findings that show added difficulty for some
students.
- Computer-based testing places more demands
on certain skills such as typing, using multiple screens to recall a
passage, mouse navigation, and the use of key combinations (Bennett, 1999;
Ommerborn & Schuemer, 2001).
- Some people become more fatigued when
reading text on a computer screen than on paper (Allan et al, 2001; Mourant,
Lakshmanan, & Chantadisai, 1981).
- Long passages may be more difficult to
read on computer screen (Haas & Hayes, 1986).
- The inability to see an entire problem on
screen at one time is challenging because some items require scrolling
horizontally and vertically to get an entire graphic on the page
(Hollenbeck, Tindal, Harniss, & Almond, 1999).
- Few teachers use computers in math
instruction, or spreadsheets, so students do not know how to “think on the
monitor” (Trotter, 2001).
- Graphic user surfaces present considerable
obstacles to students with visual impairments (Ommerborn & Schuemer, 2001).
Technological
Challenges
Computers and the Internet do not always work
the way we want them to. The word “crash” has taken on a whole new meaning in
our technology-oriented world. An issue brief of the National Governors
Association listed some of the problems: “testing sessions may be interrupted,
proceed so slowly as to interfere with student performance, or encounter
difficulties in machine operation or telecommunications that cause data to be
lost entirely. Unlike a paper-and-pencil testing system, keeping a computerized
system functioning requires significant technical expertise, which many schools
lack” (p. 7). Burk (1999) argued, “Computerized testing for students with
disabilities is viable but only with appropriate equipment, staff preparation,
and student preparation” (p. 6). Some researchers, like Hamilton, Klein, and
Lorie (2001), question whether an infrastructure currently exists that can
support the use of computers by large numbers of students. They also question
the quality of the hardware, especially with our constant evolution of
technology, and whether there is sufficient training for staff who must help
with administration and technological difficulties that may be encountered.
Also, the test program may be device-dependent; for example, there may be a
difference in contrast between monitors and speed of the computer. A test
presented online may default to the computer’s font, print size, and background
color. Graphics may become distorted on small screens, reducing standardization
of the assessment presentation. According to a report by the National Governors
Association (2002, p. 7):
The reality of statewide computerized testing is that equipment will vary from one school to the next and, sometimes, from one machine to the next within the same school. Similarly, the speed of the Internet connection may differ across schools or within the same school by time of day. The result of these variations is that one student may take a test on a small-screen monitor running at low resolution, thereby requiring repeated scrolling to read comprehension passages. Because of the Internet connection, that student may have to wait five seconds before the next passage is displayed. In contrast, another student may be able to see not only the entire passage but also the questions on the same single screen, with no wait between passages. It is known that such variations can affect performance, but it is not known how to adjust for them in test results.
A constant challenge is ongoing entry of new
Web browsers and new versions of existing browsers. In addition, HTML and
document converters are constantly being developed and modified. Unfortunately,
several features may not be universally accessible and advancements in assistive
technology are usually several steps behind new Internet components and tools.
For example, using an eye pointing device may increase the time needed to
position each eye pointing frame, leading to increased fatigue, boredom, and
inattention by the test-taker (Haaf, Duncan, Skarakis-Doyle, Carew, & Kapitan,
1999). As computer-based testing becomes a reality across states and districts,
it is important to ensure that the new technology either improves accessibility
or is compatible with existing assistive computer technology.
Security of
Online Data
Critics question whether online data are
secure. In a report by the National Governors Association (2002), security
issues related to protecting test questions and ensuring the confidentiality of
student data in a computerized system were compared to those encountered with
conventional tests and were found to be conceptually similar. Differences were
found in mechanisms to accomplish breaches and protect against them. For
example, test questions and student data could be stolen from central servers or
from local computers. This can be minimized through technical design that
encrypts questions and student records and through the careful use of passwords.
Lack of Ability
to Design Accessible Web Pages
According to WebAIM, (Web Accessibility in
Mind, an initiative of the Center for Persons with Disabilities at Utah State
University, 2001), there are 27.3 million people with disabilities who are
limited in the ways they can use the Internet: “The saddest aspect of this fact
is that the know-how and the technology to overcome these limitations already
exist, but they are greatly under-utilized, mostly because Web developers simply
do not know enough about the issue to design pages that are accessible to people
with disabilities. Unfortunately, even some of the more informed Web developers
minimize the importance of the issue, or even ignore the problem altogether” (p.
1).
Prohibitive
Development Cost
Development expenses listed in a report by
the National Governors Association (2002) include: “central hardware to deliver
the test over the Internet, local telecommunications hardware, machines in
schools for students to take the tests on, and test authoring and delivery
software. Labor expenses include costs for entering questions into the testing
software, assuring quality in the test’s operation, extracting student records
from the test database and translating the information into a form suitable for
analysis, and servicing the technology that runs the system. There are also
ongoing connection charges” (p. 7). The National Governors Association
recommends that states form consortia, cooperative agreements, or buying pools
in order to reduce the costs of “test questions, telecommunications equipment,
computer hardware, testing software, and equipment maintenance” (p. 9).
Universally Designed Computer-based Tests
Universal design is defined by the Center for
Universal Design (1997) as “the design of products and environments to be usable
by all people, to the greatest extent possible, without the need for adaptation
or specialized design.” The Assistive Technology Act of 1998 (PL 105-394)
addresses universal design through this definition:
The term ‘universal design’ means a concept or philosophy for designing and delivering products and services that are usable by people with the widest possible range of functional capabilities, which include products and services that are directly usable (without requiring assistive technologies) and products and services that are made usable with assistive technologies.
A recent report on the application of
universal design to large-scale assessments (Thompson, Johnstone, & Thurlow,
2002) found that good basic design, whether on paper or technology-based,
increases access for everyone, and poor design can have detrimental effects for
nearly everyone. Many accessibility issues relate to content and design
features, with content defined as subject matter on the page while design is
defined as the organization or arrangement of objects and information on the
page.
Content
An important function of well-designed
assessments is that they actually measure what they are intended to measure.
Test developers need to carefully examine
what is to be tested and design items that offer the greatest opportunity
for success within those constructs. Just as universally designed architecture
removes physical, sensory, and cognitive barriers to all types of people in
public and private structures, universally designed assessments need to remove
all non-construct-oriented cognitive, sensory, emotional, and physical barriers.
Assessment instructions need to be easy to
understand, regardless of a student’s experience, knowledge, language skills, or
current concentration level. Directions and questions need to be in simple,
clear, and understandable language. It is important for designers of
computer-based tests to strive for content that is understandable and navigable.
According to WebAIM (2001), “this includes not only making the language clear
and simple, but also providing understandable mechanisms for navigating within
and between pages” (p. 8).
Design Features
Legibility is the physical appearance of
text; the way shapes of letters and numbers enable people to read text “quickly,
effortlessly, and with understanding” (Schriver, 1997, p. 252). Though a great
deal of research has been conducted in this area, the personal opinions of
editors often prevail (Bloodsworth, 1993; Tinker, 1963). Bias results from items
that contain physical features that interfere with a student’s focus on or
understanding of the construct an item is intended to assess. Format dimensions
can include contrast, type size, spacing, typeface, leading, justification, line
length/width, blank space, graphs and tables, illustrations, and response
formats (see Table 2).
Table 2.
Characteristics of Maximum Legibility
| Dimension |
Maximum Legibility
Characteristics |
| Contrast |
Black type on matte pastel or off-white paper is most favorable for both
legibility and eye strain. |
| Type Size |
Large type sizes are most effective for young students who are learning to
read, students with visual difficulties, and individuals with eye fatigue
issues. |
| Spacing |
The amount of space between each character can affect legibility. Spacing
needs to be wide between both letters and words. Fixed-space fonts seem to
be more legible for some readers than proportional-spaced fonts. |
| Leading |
Leading, the amount of vertical space between lines of type, must be enough
to avoid type that looks blurry and has a muddy look. The amount needed
varies with type size (for example, 14-point type needs 3-6 points of
leading). |
| Typeface |
Standard typeface, using upper and lower case, is more readable than italic,
slanted, small caps, or all caps. |
| Justification |
Unjustified text (with staggered right margin) is easier to see and scan
than justified text – especially for poor readers. |
| Line Length |
Optimal length is about 4 inches or 8 to 10 words per line. This length
avoids reader fatigue and difficulty locating the beginning of the next
line, which causes readers to lose their place. |
| Blank Space |
A general rule is to allow text to occupy only about half of a page. Blank
space anchors text on the paper and increases legibility. |
| Graphs and Tables |
Symbols used on graphs need to be highly discriminable. Labels should be
placed directly next to plot lines so that information can be found quickly
and not require short-term memory. |
| Illustrations |
When used, an illustration should be directly next to the question for which
it is needed. Because illustrations create numerous visual and distraction
challenges, and may interfere with the use of some accommodations (such as
magnifiers), they should be used only when they contain information being
assessed. |
Response Formats |
Response options should include larger circles (for bubble response tests),
as well as multiple other forms of response. |
From Thompson, Johnstone, & Thurlow, 2002.
It is important to maintain these aspects of
universal design when converting paper/pencil tests to computer-based tests.
Poor design on paper will result in poor design on a screen. In addition to the
universal design elements described above, computer-based testing can offer
several additional features that can increase the accessibility of assessments
for all students, including students with disabilities and English language
learners. According to WebAIM (2001), “Everyone benefits from well-designed Web
sites, regardless of cognitive capabilities. In this context, ‘well-designed’
can be defined as having a simple and intuitive interface, clearly worded text,
and a consistent navigational scheme between pages” (p. 8). These features also
need to take into account variations in technology available in schools across a
district or state, and the other challenges described in the previous section.
The provision of navigation tools and
orientation information in pages can maximize access for all users. However,
there are users who cannot access visual clues such as image maps, scroll bars,
side-by-side frames, or graphics. Some users lose contextual information because
they are accessing a page one word at a time through speech synthesis or
braille. Ommerborn and Schuemer (2001, p. 21) conducted a survey of German
students with disabilities and found that:
Being able to use various ways of sending commands within a programme not only helps people with specific handicaps, but also renders working with a computer much more comfortable for all users with their different preferences and skills…Multimedia products addressing several senses or allowing the user to choose between visual and acoustic information not only makes access easier for people with impaired senses but also makes the product altogether more attractive.
Assistive Technology
Even though items on universally designed
assessments will be accessible for most students, there will still be some
students who continue to need accommodations, including assistive technology.
According to Bowe (2000), “One big advantage of universal design is that it
minimizes the need, on the part of people with disabilities, for assistive
technology devices and services” (p. 25). Items are biased when they do not
allow for adaptation for use with assistive technology that is needed to
facilitate use of the student’s primary means of communication. Computer-based
tests need to be accessible for a variety of forms of assistive technology
(e.g., key guards, specialized keyboards, trackballs, screen readers, screen
enlargers) for students with physical or sensory disabilities. Bowe (2000)
stated, “If a product or service is not usable by some individual, it is the
responsibility of its developers to find ways to make it usable, or, at minimum,
to arrange for it to be used together with assistive technologies of the user’s
choice” (p. 27). Appendix A describes several resources to assist assessment
developers in increasing access to assistive technology.
It is important to note that making
computer-based testing amenable to assistive technology does not mean that
students will automatically know what to do. Educators, especially special
educators, need to be competent in technology knowledge and use. According to
Lahm and Nickels (1999), “Educators must become proactive in their
technology-related professional development because teacher education programs
have only recently begun addressing the technology skills of their students” (p.
56). The Knowledge and Skills Subcommittee of the Council for Exceptional
Children’s (CEC) Professional Standards and Practice Standing Committee has
developed a set of 51 competencies for assistive technology that cross 8
categories, along with knowledge and skills statements for each category (see
Lahm & Nickels, 1999).
Laws Governing
Assistive Technology
The use of assistive technology is defined in
the Individuals with Disabilities Education Act (IDEA 97), the Rehabilitation
Act of 1997, and is implied in the Americans with Disabilities Act (ADA). IDEA
97 defines assistive technology as “any item, piece of equipment, or product
system…that is used to improve the functional capabilities of individuals with
disabilities; and any service that directly assists an individual in the
selection, acquisition, or use of an assistive technology device.” An “assistive
technology device” is further defined as “any item, piece of equipment, or
product system, whether acquired commercially off the shelf, modified, or
customized, that is used to increase, maintain, or improve the functional
capabilities of a child with a disability” (20 U.S.C. 1401(1)).
The Rehabilitation Act (reauthorized in 1997)
requires institutions receiving federal funds to have accessible Web sites.
Similarly, the Americans with Disabilities Act (ADA) requires covered entities
to furnish appropriate auxiliary aids and services where necessary to ensure
effective communication with individuals with disabilities, unless doing so
would result in a fundamental alteration to the program or service or in an
undue burden (See 28 C.F.R. 36.303; 28 C.F.R. 35.160). Auxiliary aids include
taped texts, Brailled materials, large print materials, captioning, and other
methods of making audio and visual media available to people with disabilities.
Titles II and III of the ADA require State and local governments and the
business sector to provide effective communication whenever they communicate
through the Internet. In order to specifically address the needs of people with
visual disabilities, an ADA policy ruling determined that a text format rather
than a graphical format assures accessibility to the Internet for individuals
using screen readers. Without special coding, a text browser will only display
the word “image” when it reads a graphic image, and if the graphic is essential
to navigating the site (e.g., navigational button or arrow) or if it contains
important information (e.g., table or image map) the user can get stuck and not
be able to move or understand the information provided.
Assistive
Technology Resources
There are several resources available to increase the accessibility of computer-based testing for students with disabilities. These resources are found primarily in the area of general Web content. Chishold, Vanderheiden, and Jacobs (1999) offer guidelines on how to make Web content accessible to people with disabilities. They are quick to point out that following these guidelines can also make Web content more available to all users, including those who use voice browsers, mobile phones, automobile-based personal computers, and other technology. The guidelines, found in Table 3, explain how to make multimedia content more accessible to a wide audience. For more information about Web accessibility, visit http://www.webaim.org, the official Web site of Web Accessibility in Mind (WebAIM). Several additional resources can be found in Appendix A.
Table 3. Web Content Accessibility Guidelines
21
December 2001:
The Authoring Tool Accessibility Guidelines Working Group has released the first
public Working Draft of Authoring Tool Accessibility Guidelines "Wombat". The
guidelines are for developers who wish to design authoring tools that produce
accessible Web content and who wish to create accessible authoring interfaces)
Guideline 1. Provide equivalent
alternatives to auditory and visual content.
Provide content that, when presented to the user, conveys essentially the same
function or purpose as auditory or visual content.
Guideline 2. Don't rely on color alone.
Ensure that text and graphics are understandable when viewed without color.
Guideline 3. Use markup and style sheets
and do so properly.
Mark up documents with the proper structural elements. Control presentation with
style sheets rather than with presentation elements and attributes.
Guideline 4. Clarify natural language
usage.
Use markup that facilitates pronunciation or interpretation of abbreviated or
foreign text.
Guideline 5. Create tables that
transform gracefully.
Ensure that tables have necessary markup to be transformed by accessible
browsers and other user agents.
Guideline 6. Ensure that pages featuring
new technologies transform gracefully.
Ensure that pages are accessible even when newer technologies are not supported
or are turned off.
Guideline 7. Ensure user control of
time-sensitive content changes.
Ensure that moving, blinking, scrolling, or auto-updating objects or pages may
be paused or stopped.
Guideline 8. Ensure direct accessibility
of embedded user interfaces.
Ensure that the user interface follows principles of accessible design:
device-independent access to functionality, keyboard operability, self-voicing,
etc.
Guideline 9. Design for
device-independence.
Use features that enable activation of page elements via a variety of input
devices.
Guideline 10. Use interim solutions.
Use interim accessibility solutions so that assistive technologies and older
browsers will operate correctly.
Guideline 11. Use W3C technologies and
guidelines.
Use W3C technologies (according to specification) and follow accessibility
guidelines. Where it is not possible to use a W3C technology, or doing so
results in material that does not transform gracefully, provide an alternative
version of the content that is accessible.
Guideline 12. Provide context and
orientation information.
Provide context and orientation information to help users understand complex
pages or elements.
Guideline 13. Provide clear navigation
mechanisms.
Provide clear and consistent navigation mechanisms -- orientation information,
navigation bars, a site map, etc. -- to increase the likelihood that a person
will find what they are looking for at a site.
Guideline 14. Ensure that documents are
clear and simple.
Ensure that documents are clear and simple so they may be more easily
understood.
Computerized Adaptive Testing
In computerized adaptive testing, a student
responds to an item, which is followed by more difficult items if the student
responded correctly, or easier items if the student responded incorrectly
(Hamilton, Klein, & Lorié, 2001). Through this process, a student’s performance
level is determined. According to Hamilton, Klein and Lorié (2001), “each
response leads to a revised estimate of the student’s proficiency and a decision
either to stop testing or to administer an additional item that is harder or
easier than the previous one” (p. 12).
The advantages cited for computerized
adaptive testing include short and efficient administration time, with the
computer selecting the next item immediately after an item is completed. A
proficiency level is determined through the completion of fewer items than a
test in which students respond to every item on the test. According to McBride
(1985), “A well-constructed adaptive test attains a specified level of
measurement precision in about half the length of time a conventional test would
require to reach the same level. This is attributable to the adaptive feature;
by tailoring the choice of questions to match the examinee’s ability, the test
bypasses most questions that are inappropriate in difficulty level and
contribute little to the accurate estimation of the test-taker’s ability” (p.
26).
However Stone and Lunz (1994) found that the
inability of students taking computerized adaptive tests to review items and
alter their responses may affect the quality of measurement. Students cannot
select the order in which they respond to items, or leave some items blank.
There is some research that suggests that
students who change earlier answers may improve their scores by a small margin
(Gerson & Bergstrom, 1995; Stocking, 1996). There is also concern that some
students may respond to early items wrong on purpose to get easier questions
(Wainer, 1993).
The use of computerized adaptive tests for
large-scale assessments has come under scrutiny by federal officials who
question whether “levels” testing meets accountability requirements of Title I
(Olson in Education Week, 2002). Levels testing,
which has been defined as testing at a student’s instructional level rather than
at his or her grade level, relies on overlapping levels within a single grade
level, and common items among the levels. Computerized adaptive testing goes
beyond the need for separate booklets by using a variety of complex algorithms
that allows the student to move among different “levels” more freely, based on
performance (Quenemoen, Thurlow, & Bielinski, in press).
Process for
Developing Inclusive Computer-based Tests
The transformation of traditional
paper/pencil tests to inclusive computer-based tests takes careful and thorough
work that includes the collaborative expertise of many people. As discussed
earlier in this paper, in order for the full benefits of computer-based testing
to be realized, a thoughtful and systematic process to examine the transfer of
existing paper/pencil assessments must occur. It is not enough to simply
transfer test items from paper to screen. Not only will poor design elements on
the paper test transfer to the screen, additional challenges may result in
reducing the validity of assessment results. Some of the challenges
traditionally present with accommodations could be minimized through universally
designed computer-based tests, while others might remain or present even greater
challenges. Here are some steps to follow in addressing these transformation
issues.
Step 1. Assemble a group of experts to guide the transformation. This group needs to include experts on assessment design, accessible Web design, universal design, and assistive technology, along with state and local assessment and special education personnel. Table 4 contains a worksheet to use when gathering this group.
Table 4. Assemble a Group of Experts to Guide the Development of Computer-based
Tests.
Type of Expert |
Names/Positions |
Assessment design experts |
|
Accessible Web design experts |
|
Universal design experts |
|
Assistive technology experts |
|
State assessment personnel |
|
State special education personnel |
|
Local assessment personnel |
|
Local special education personnel |
|
Step 2.
Decide how each accommodation will be incorporated into the computer-based test.
Examine each possible accommodation in light of computer-based administration.
Some of the traditional paper/pencil accommodations will no longer be needed
(e.g., marking responses on test form rather than on answer sheet), while others
will become built-in features that are available to every test-taker. Some
accommodations will be more difficult to incorporate than others, requiring
careful work by test designers and technology specialists. The standards and
guidelines for accessible Web design found in Appendices B, C, and D should be
used when building in these features.
Step 3.
Consider each accommodation or assessment feature in light of the constructs
being tested. For example, what are the implications of the use of a screen
reader when the construct being measured is reading, or the use of a spellcheck
when achievement in spelling is being measured as part of the writing process?
As the use of speech recognition technology permeates the corporate world,
constructs that focus on writing on paper without the use of a dictionary or
spellchecker may become obsolete and need to be reconsidered.
Step 4.
Consider the feasibility of incorporating the accommodation into computer-based
tests.
Questions about the feasibility of the accommodation may require review by
technical advisors, or members of a policy/budget committee, or may require
short-term solutions along with long term planning. According to the Technology
Act of 1998 (§ 1194.2 Application):
(a) When developing, procuring, maintaining, or using electronic and information technology, each agency shall ensure that the products comply with the applicable provisions of this part, unless an undue burden would be imposed on the agency.
(1) When compliance with the provisions of this part imposes an undue burden, agencies shall provide individuals with disabilities with the information and data involved by an alternative means of access that allows the individual to use the information and data.
Construct a specific plan for building in
features that are not immediately available, in order to keep them in the
purview of test developers. Extensive pilot testing needs to be conducted with a
variety of equipment scenarios and accessibility features.
Step 5.
Consider training implications for staff and students. The best technology
will be useless if students or staff do not know how to use it. Careful design
of local training and implementation needs to be part of the planning process.
Special consideration needs to be given to the computer literacy of students and
their experience using features like screen readers. Information about the
features available on computer-based tests needs to be marketed to schools and
available to IEP teams to use in planning a student’s instruction and in
preparation for the most accessible assessments possible. Practice tests that
include these features need to be available to all schools year around. This
availability presents an excellent opportunity for students whose schools have
previously been unaware of or balked at the use of assistive technology.
Considerations
Most states have a list of possible or common
accommodations for students with disabilities within the categories of
timing/scheduling, presentation, response, and setting (Thurlow, Lazarus, &
Thompson, 2002). Some states also list accommodations specifically designed for
students with limited English proficiency (Rivera, Stansfield, Scialdone, &
Sharkey, 2000).
Presentation
Accommodations
The list of accommodations in Table 5 is an
expanded list of presentation accommodations generated to address the needs of
students with a variety of accommodation needs—including students with
disabilities, students with limited English proficiency, students with both
disabilities and limited English proficiency, and students who do not receive
special services, but have a variety of unique learning and response styles and
needs. For each accommodation, relevant considerations are provided in the
table. The three columns to the right of the Considerations Column represent:
- A built-in feature of universally designed
computer-based tests (available for self-selection by any student)
- The need for this accommodation is not
affected by computer-based testing
- A new or different accommodation may be
needed for computer-based testing
Following Table 5 is a summary of considerations for each of the presentation accommodations.
Table 5. Presentation Accommodations
| Accommodation |
Considerations for Computer-based Tests |
1* |
2* |
3* |
| Large print and magnification
|
Capacity for any student to
self-select print size or magnification
Graphics and text-based user interfaces have different challenges Scrolling issues Determination of optimal
print size (e.g., default set at 14 pt) and size of graphics to reduce need
for large print or magnification Variations in screen
size Effects of magnification
on graphics and tables |
X |
|
X |
| Instructions simplified/clarified
|
Instructions designed for maximum
simplicity/clarity
Capacity for student to self-select alternate versions of instructions in
written or audio format Capacity to have
instructions repeated as often as student chooses Variable audio speed |
X |
|
|
| Audio presentation of
instructions and test items
|
Capacity for any student to
self-select audio (screen reader) presentation of instructions (all students
wear ear/headphones) Graphics and text-based user
interfaces have different challenges Capacity to repeat
instructions and items as often as student chooses Variable audio speed Audio presentation must
be high quality |
X |
|
|
| Instructions and test items
presented in sign language
|
Capacity for student to
self-select alternate versions of instructions in written format Capacity for student to
self-select signed versions of instructions and test items (Note: some words
may not be easily translated into sign language) Graphics and text-based
user interfaces have different challenges Not feasible to read
lips on video |
X |
|
|
| Instructions and test items
presented in a language other than English
|
Capacity for student to
self-select alternate language versions of test items in written or audio
format Beware of the speed at which some
languages are produced, they may take more space than English Machine translation
capability Graphics and text-based
user interfaces have different challenges Capacity for pop-up
translation Variable audio speed |
X |
X |
|
| Braille |
Use of screen reader that converts text into synthesized speech or Braille Alternative tags for images Graphics and text-based
user interfaces have different challenges Students using Braille
may require extra time |
|
X |
X |
| Highlighter and place holding
templates |
Capacity for any student to
self-select highlighter
Graphics and text-based user interfaces have different challenges Clear instructions for
use of highlighter |
X |
|
|
| Graphics or images that
supplement text |
Careful selection of images Alternative text or "alt tags" for images Graphics and text-based
user interfaces have different challenges Avoidance of complex
backgrounds or wallpaper that may interfere with the readability of
overlying text Tactile graphics or
three-dimensional models may be needed for images |
|
|
X |
| Paper/pencil test format |
Students who are not computer
literate Students who need accommodations
that are not available on computer-based assessments |
|
|