|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Including English
Language Learners
with Disabilities in
Large-Scale
Assessments: A Case
Study of
Linguistically-Diverse
Populations
ELLs with Disabilities Report 14Published by the National Center on Educational OutcomesPrepared by: July 2006 Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as: Minnema, J. E., Thurlow, M. L., VanGetson, G. R., & Jimenez, R. (2006). English language learners with disabilities in large-scale assessments: A case study of linguistically-diverse populations (ELLs with Disabilities Report 14). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/ELLsDisReport14.html Overview The advent of standards-based reform during the past 10 years has ushered in a variety of challenges for policymakers and practitioners alike. Such concerns were accentuated by the legislative mandates of the No Child Left Behind (NCLB) Act which required year-to-year academic performance to be measured by states’ standards-based large-scale assessments for all subgroups of students in U.S. schools. While schools and their staff generally support the theory of action that underlies initiatives like NCLB, such as that explicated in Testing, Teaching, and Learning (Elmore & Rothman, 1999), it is still a challenge to provide instruction based on challenging, grade-level content standards. In addition, there is often a concern that some students, especially those with disabilities or limited English proficiency, may not be capable of achieving the academic content deemed appropriate for the grades in which they are enrolled in school. It is also suggested that these students are not able to fully participate in large-scale assessments that were designed for their peers. While concerns have been raised for students with disabilities as a subgroup and English language learners (ELLs) as a subgroup for some time, it is only recently that including students with both disabilities and English learning challenges in states’ large-scale assessment and accountability programs have been considered. With the requirements of the Individuals with Disabilities Education Act (IDEA) and NCLB, both of which clearly required that students with disabilities be included in state assessments, states are making progress toward including all students in their standards-based testing. Rates of participation for students with disabilities and English language learners have been improving over time (Thompson & Thurlow, 2003). However, there are few data that can demonstrate improved academic results for ELLs with disabilities. In fact, there are few sources of public data that report results for these students (Albus & Thurlow, 2005). Only recently, ELLs with disabilities have begun to receive marginal attention in the literature (Minnema, Thurlow, Anderson, & Stone, 2005). The National Center on Educational Outcomes (NCEO) has for the past four years conducted research on large-scale assessment and instructional issues for ELLs with disabilities, but neither NCEO’s research nor any other research study has yet described large-scale assessment experiences at the local school level for English language learners with disabilities. This study was designed, in part, to clarify some of the issues that surround including English language learners in states’ large-scale assessment programs. Specifically, we gathered practical information at the local school level to understand these students’ large-scale assessment experiences from a variety of perspectives, to describe the characteristics of ELLs with disabilities as well as the characteristics of their schools, and to make known the level of awareness that students and their families have about large-scale assessments. State and Local ContextTo understand the school and students included in this study, it is helpful to know the context and characteristics of the state and the district in which the study took place. The state was a large southern state with a total estimated population of 17,019,068 in 2003. The breakdown by ethnicity was 65.4% White not of Hispanic/Latino origin, 16.8% of Hispanic or Latino origin, 14.6% Black/African American, 1.7% Asian, 0.3% American Indian or Alaska Native, 0.1% Native Hawaiian or Other Pacific Islander, 3.0% reporting some other race, and 2.4% reporting two or more races. Across these subgroups, 16.7% (n = 2,670,794) were foreign born. The regions in which these individuals were born were Latin America (72.8%, a total of 1,943,781), Europe (13.3%, n = 355,427), Asia (8.7%, n = 231,976), Africa (1.3%, n = 34,495), Northern America excluding the United States (3.8%, n = 100,158), and Oceania (0.2%, n = 4,957). Of the 15,043,603 individuals in this state who are five years old or older, 23.1% (n = 3,473,864) spoke a language other than English at home. In 2003, a total of 397,758 students, or 9.6% of school age individuals (ages 3 through 21 years) received special education services under the Individuals with Disabilities Education Act (IDEA; U.S. Office of Special Education Programs, 2003). All demographic information was gained from the 2000 Census unless otherwise indicated (U.S. Census Bureau, 2005). The School District The district was an urban school district located in a large metropolitan area in the southern region of the state. The district served approximately 365,000 students from Pre-Kindergarten through grade 12 in a total of 432 school buildings. Students attending school in this district were predominately of Hispanic or Latino origin (58%), Black (29%), or White (10%). About 1% of the students were Asian, and less than 1% were American Indian or Multi-Racial. About 17% of students in the district were classified as limited English proficient, speaking a total of 106 different languages. In addition, 12% of the students in the district received special education services. A total of 64% of district students qualified for the federally funded free or reduced price lunch program. Large-Scale Assessment Program The assessment program for this state helps track student performance based on the state’s content standards. Students in grades 3 through 10 take a standards-based assessment in math and reading. A writing test is administered in grades 4, 8, and 10 and a science test is administered in grades 5, 8, and 11. The assessment system is considered high-stakes because all high school students must pass the grade 10 test in reading and math in order to graduate, and all grade 3 students who score at level 1 on the reading test must repeat third grade. Reporting of the test results uses a 5-point scale of proficiency, with level 1 indicating the lowest demonstrated proficiency level and level 5 indicating the highest. The writing test is scored on a scale of 1-6. The state’s Department of Education rates each school using a letter grade (A-F); the rating is based on overall student performance, the percentage of eligible students who took the test, and evidence of student progress in reading and math. The Elementary Schools The first elementary school (School 1) serves approximately 1,300 students in Pre-Kindergarten through grade 5. The school offers an extensive, holistic, dual-language instructional program. Through this program, all students receive instruction in language arts, science, social studies, and mathematics in English and Spanish. Visual arts, music, and physical education classes are taught in English. The school is located in a special neighborhood of the metropolitan area where the majority of the students (87%) are of Hispanic or Latino origin, while 9% are White, 1% are Black, and 2% are classified as Asian/Multi-racial. The majority of students (71%) participate in the free or reduced price lunch program. Approximately 22% of the student body is limited English proficient. In 2004–2005, School 1 was ranked an "A" school. The other elementary school (School 2) educates approximately 1,200 students in Pre-Kindergarten through grade 5. The majority of the students are of Hispanic or Latino origin (91%), 4% of the students are Black, 4% are White, and less than 1% are Asian or Multi-Racial. Ninety percent of the students qualify for the free or reduced price lunch program, and 52% are classified as limited English proficient. School 2 was ranked an "A" school in 2004–2005. The Middle School School 3 serves approximately 2,100 students in grades 6 through 8. The students are predominately of Hispanic or Latino origin (80%), followed by Black (12%) and White (7%). Less than 1% are Asian or Multi-Racial. About 76% of the student body qualifies for free or reduced price lunch and about 16% are classified as limited English proficient. In 2004–2005, School 3 was ranked a "C" school. The High School School 4 serves approximately 2,300 students in grades 9 through 12. The school is located in another of the metropolitan area’s special neighborhoods, and serves predominately Black students (95%). A small portion of the students are Hispanic (5%), and less than 1% are White, Asian, or Multi-Racial. Approximately 63% of the student body qualifies for free or reduced price lunches and 19% are classified as limited English proficient. School 4 was ranked an "F" school in 2004–2005. MethodResearch Questions We addressed four research questions in our study: 1) What perceptions do educators, parents, and students have about the experiences of English language learners with disabilities who participate in large-scale assessments? 2) How are English language learners with disabilities performing in large-scale assessments? 3) How are participation decisions made to test English language learners with disabilities in large-scale assessments? 4) What are the characteristics of schools that test English language learners with disabilities in large-scale assessments? Research Design Our collective case study research design, in which one school is defined as a case, used a mixed method approach to collect quantitative and qualitative data from four sources of data. Data were collected on site in the four schools as well as district offices. Sample Using a purposive sample, we included English language learners with disabilities (n = 27), their parents (n = 27), school level educators and administrators (n = 60), and district level administrators (n = 2). The schools from which our sample was drawn were recruited by the district’s Assistant Superintendent of Special Education and the Program Specialist of Special Education/English Language Learners. Within each school, one staff member served as a contact person for the study. Their primary responsibilities were to recruit parents and students for the face-to-face interviews and help with the administration of the educator surveys. All interview participants received a gift card from a department store as a thank you for their time invested in our research activities. Instruments We used a variety of self-developed data collection instruments that included a written survey, interview protocols, and a document review data collection sheet (see the Appendix for copies of the survey and interview protocols). Procedures The written surveys were distributed in three ways. In School 1, we mailed the surveys to the contact person who distributed and collected them over the course of a month. In School 2, the surveys were placed in teachers’ school mailboxes with a request to return the surveys to the contact person at that school by the end of that week. Surveys were not distributed in School 3 at the school’s request. In School 4, the surveys were given to the contact person at the school to distribute and then collect from teachers. After collecting all the completed surveys from each school, the contact person at each school returned the surveys to us via a self-addressed, postage-paid envelope. Face-to-face interviews with parents were conducted at school in either small groups or individually depending on the participants’ preferences. Each interview was recorded for subsequent transcription and data analysis. An English speaking researcher conducted all of the interviews with or without an interpreter. All of the parent interviews were conducted in the parent’s self-reported dominant language: English, Spanish, or Haitian-Creole. When the parent’s dominant language was Spanish, a bilingual researcher of Latino origin interpreted for the English-speaking researcher and the parent. When the parent’s dominant language was Haitian-Creole, a Haitian bilingual district employee interpreted for the English-speaking researcher and the parent. One researcher interviewed the students in School 4 in English during noninstructional time. Since the student interview responses were shorter than the parent narrative data, these interviews were not recorded. Instead, student responses were written down during the interviews. Depending on the size of the group, parent interviews required from 30 to 45 minutes to complete with individual interviews requiring less time. Student interviews were typically less than 10 minutes in duration. Educator interviews were conducted via telephone due to time constraints of the study. Educators were identified through each school’s contact person. All educator interviews were conducted in English by one researcher. Each interview was recorded to be transcribed at a later date and lasted approximately 30–45 minutes. Two researchers collected data for the document review of students’ Individualized Education Programs (IEPs) and cumulative files. A common data collection sheet was used across schools, collecting data on a number of variables, including English language services (e.g., description of services, language assessment scores), special education services (e.g., type of services, content areas, language of instruction), country of origin, time in the U.S., prior school history, attendance, disciplinary issues, and grades. The data source (e.g., IEP, cumulative file) was also indicated for each variable. Data Analysis To analyze our narrative data, we first transcribed all English portions of the educator and parent interviews verbatim. For the Spanish portions of our parent interviews, the bilingual researcher transcribed the Spanish data and then translated these data to English. For the Haitian-Creole portions of our parent interviews, the Haitian bilingual school district employee transcribed the Haitian-Creole data and then translated these data to English. All narrative data were then subjected to a content analysis that yielded themes of results. Throughout the qualitative analysis process, English data were compared back to Spanish and Haitian-Creole translated data to ensure accuracy of our interpretations. We used the original Spanish and Haitian-Creole as well as the translated English for any supportive quotations taken from the parent interview data. To analyze the student interview data, we tabulated categories of responses rather than creating themes of results. We employed descriptive statistics to analyze the document review and survey data. ResultsWritten Survey: Summary Across Schools The written survey data were first compiled across Schools 1, 2, and 4 since surveys were not completed at School 3. A total of 60 (n = 60) teachers and administrators completed the survey. The 21 survey items that were included in analysis were organized into four categories: (1) participation-related data elements, (2) performance-related data elements, (3) student and parent-related data elements, and (4) teacher-related data elements. Each category is presented separately with summary text and figures representing the frequencies and percentages of responses reported for each item. Survey items included the proper name of the state test, but to preserve the anonymity of the district, the large-scale assessment is referred to as the "state test." Also, the term "blank" indicates "no response" to that item. Participation-Related Data Elements. Most of the participants indicated that English language learners with disabilities always take the state test and always use accommodations when taking the test (items 1 and 2). These accommodations are more likely to be designed for special education than for second language; 62% of the respondents indicated that these students usually or always use only special education accommodations to take the state tests while only 30% indicated that these students usually or always use only second language accommodations to take the state tests (items 3 and 4). Combining these two types of accommodations groups may be more popular than using either one alone as 71% of respondents reported that these students always or usually use both special education and second language accommodations (item 5). Further, most participants indicated that English language learners with disabilities hardly ever take an alternate assessment instead of the regular state test (item 6). In terms of test completion, educators reported that these students typically complete all of the state test items, sometimes complete about half of the state test items, and rarely or never complete 10 or less of the state test items (items 7–9). Results from participation-related survey items are shown in Figure 1. Figure 1. Participation-Related Data Elements 1) English language learners with disabilities take the state test. Count Percent 2) English language learners with disabilities use accommodations to take the state test. Count Percent 3) English language learners with disabilities use only special education accommodations to take the state test. Count Percent 4) English language learners with disabilities use only second language accommodations to take the state test. Count Percent 5) English language learners with disabilities use both special education and second language accommodations to take the state test. Count Percent 6) Most of the English language learners with disabilities in my school participate in an alternate assessment to the state test. Count Percent 7) English language learners with disabilities complete all state test items. Count Percent 8) English language learners with disabilities complete about half of the state test items. Count Percent 9) English language learners with disabilities complete ten or less state test items Count Percent Performance-Related Data Elements. Participants had varied responses when asked whether English language learners with disabilities can demonstrate what they know and can do on the state tests. While 7% responded that these students are never able to demonstrate their abilities on the state test, the majority of respondents were more optimistic about student performance—55% responded that these students usually or always demonstrate what they know and can do on the state test (item 10). Educators responded consistently when asked about proficiency on and passing the state test, and the most frequent answer was "rarely." These students rarely can be proficient and rarely are proficient on the state test; they rarely can pass and rarely do pass the state test (items 11–14). Results from performance-related survey items are presented in Figure 2. Figure 2. Performance-Related Data Elements 10) English language learners with disabilities can demonstrate what they know and can do on the state test. Count Percent 11) English language learners with disabilities can be proficient on the state test. Count Percent 12) English language learners with disabilities are proficient on the state test. Count Percent 13) English language learners with disabilities can pass high stakes tests. Count Percent 14) English language learners with disabilities do pass high stakes tests. Count Percent Student- and Parent-Related Data Elements. The majority of the respondents reported that parents of ELLs with disabilities typically understand what the state test is and what test accommodations are (items 15 & 16). Similarly, respondents indicated that ELLs with disabilities usually or always understand what the state test is and what test accommodations are (items 17& 18). Results from student and parent-related survey items are shown in Figure 3. Figure 3. Student- and Parent-Related Data Elements 15) English language learners with disabilities’ parents understand what the state test is. Count Percent 16) English language learners with disabilities’ parents understand what test accommodations are. Count Percent 17) English language learners with disabilities understand what the state test is. Count Percent 18) English language learners with disabilities understand what test accommodations are. Count Percent Teacher-Related Data Elements. Educators were fairly confident that there is someone with second language expertise on the IEP teams for English language learners with disabilities (item 19). Of the IEP team members, respondents reported that special education teachers primarily decide what test accommodations these students will use to take the state test, followed by second language teachers, other individuals, and general education teachers (item 20). Participants indicated that these educators are also the most likely to know the family background of ELLs with disabilities, along with the students’ parents (item 21). Results from teacher-related survey items are presented in Figure 4. Figure 4. Teacher-Related Data Elements 19) There is someone with second language expertise on English language learners with disabilities IEP teams in my school. Count Percent 20) Who decides what test accommodations English language learners with disabilities use to take the state test? Count Percent 21) In your school, who knows English language learners with disabilities family background information? Count Percent
To gain a different perspective on the data and facilitate making comparisons between schools, results are reported in this section on a school-to-school basis. The same grouping categories and reporting strategies are used here as in the previous section. A total of 18 (n = 18) educators from School 1, 30 (n = 30) educators from School 2, and 12 (n = 12) educators from School 4 completed the survey. Again, Schools 1 and 2 are elementary schools while School 4 is a high school. Participation-Related Data Elements. In both Schools 1 and 2, a large majority of the educators indicated that ELLs with disabilities always take the state test and always use accommodations when they take the test. In comparison, the majority of the educators at School 4 indicated that ELLs with disabilities usually take the state test and usually use accommodations when they take the test (items 1 and 2). Educators from the elementary and high school levels differed in their responses regarding the type of accommodations used. Participants from Schools 1 and 2 responded that English language learners with disabilities are more likely to use only special education accommodations than second language accommodations when taking the state test. The majority of the participants from School 4 responded that these students rarely use only special education or second language accommodations (items 3 and 4). When asked whether these students use both special education and second language accommodations to take the state tests, educators from School 1 overwhelmingly reported that this is always true, while educators from the other schools were less sure, and mostly responded "usually" to this item (item 5). When asked about an alternate assessment, participants from School 1 responded that ELLs with disabilities hardly ever participate in an alternate assessment instead of the state test, while participants from School 2 mostly responded that these students rarely participate in an alternate assessment, and educators from School 4 responded that these students usually participate in an alternate assessment to the state test (item 6). Finally, regarding state test completion, respondents from Schools 1 and 4 reported that ELLs with disabilities usually complete all of the state test items while respondents from School 2 responded equally among the "always," "usually," and "rarely" answers (item 7). Elementary educators responded consistently to items asking whether these students complete about half or 10 or less of the state test items; they responded that these students rarely, if ever, complete about half of the test items or complete 10 or less items. School 4 participants continued with their "usually" response selection, indicating that these students usually complete about half or 10 or less of the state test items (items 8 and 9). Results from participation-related survey items by school are presented in Figure 5. Figure 5. Participation-Related Data Elements 1) English language learners with disabilities take the state test.
3) English language learners with disabilities use only special education accommodations to take the state test.
4) English language learners with disabilities use only second language accommodations to take the state test.
5) English language learners with disabilities use both special education and second language accommodations to take the state test.
7) English language learners with disabilities complete all state test items.
8) English language learners with disabilities complete about half of all state test items.
9) English language learners with disabilities complete ten or less of all state test items.
Figure 6. Performance-Related Data Elements 10) English language learners with disabilities can demonstrate what they know and can do on the state test.
11) English language learners with disabilities can be proficient on the state
test.
13) English language learners with disabilities can pass high stakes tests.
Student- and Parent-Related Data Elements. Educators from different schools responded differently when asked about English language learners with disabilities’ and their parents’ understanding of the state test and test accommodations. In School 1, educators indicated these students always understand what the state test is, and always understand the concept of test accommodations. They were less convinced of the parents’ knowledge of the state test and test accommodations, indicating that parents only usually understand what the state test is and the concept of test accommodations. Educators at School 2 did not converge on one response regarding ELLs with disabilities’ and their parents’ understanding of the state test and accommodations, but reported very few "never" responses. Educators at School 4 reported that parents rarely grasp the concept of the state test or test accommodations (items 15 and 16), but the students usually understand what the state test is and what test accommodations are (items 17 and 18). The results from the student and parent-related survey items by school are shown in Figure 7.Figure 7. Student- and Parent-Related Data Elements 15) English language learners with disabilities’ parents understand what the state test is.
16) English language learners with disabilities’ parents understand what test accommodations are.
17) English language learners with disabilities understand what the state test is.
18) English language learners with disabilities understand what test accommodations are.
Teacher-Related Data Elements. The participants from both elementary schools reported that there is always someone with second language expertise on English language learners with disabilities’ IEP teams. Educators at the high school were less sure about the presence of an individual with second language expertise on these students’ IEP teams, reporting that this individual is usually, but not always, present. Educators from all three schools indicated that special education teachers typically make the decision about what test accommodations these students will use to take the state test. Second language teachers, general education teachers, and other individuals also contribute to this decision. Participants were also asked to indicate which individuals in the school were most familiar with the family background of English language learners with disabilities. At Schools 1 and 4, many educators did not know who knew the most about the family backgrounds. Other frequent responses by educators at Schools 1 and 4 were the same as these responses provided by educators at School 2: special education, second language, and general education teachers were most familiar. The nature of these data did not agree with a simple graphical representation of the data showing between school comparisons; therefore, these data are not represented by a figure.Document Review Results from the document review are presented by school to better reflect individual student information. Data for nine students from School 1, six students from School 2, five students from School 3, and seven students from School 4 are presented. Student information reported includes grade level, age, country of origin, years lived the in U.S., disability label, and retention status. Student assessment and services information reported includes large-scale assessment information (e.g., accommodations, alternate assessment, exempt status), language assessment date and the resulting English language level assigned by the school, second language instructional services provided, and special education services provided to the student. School 1. Four of the nine students attended the school’s pre-Kindergarten program, and ranged from four to five years of age; there were also two grade 3 students (ages 10 and 12), one grade 1 student (age 6), one grade 2 student (age 8), and one grade 5 student (age 13). Most of these students were born in the United States, with the exception of student 3 (Puerto Rico), student 7 (Cuba), and student 9 (Peru). Of the foreign-born students, student 3 has resided in the U.S. for three years, most of his or her life, while student 7 resided in the U.S. for four years, only one-third of his or her life. It is unknown how long student 9 had been in the U.S. These students’ disability labels included specific learning disability (SLD; n = 6), speech impaired (SI; n = 5), developmentally delayed (DD; n = 3), language impaired (LI; n = 4), and other health impaired (OHI; n = 1). Five of the nine students had multiple disability labels. In terms of retention status, students 6, 7, and 9 were retained in Kindergarten, grade 3, and grade 2, respectively. Table 1 presents the information for the nine students at School 1. Table 1. Student Information for School 1
Students below grade 3 did not have any large-scale assessment information in their files except for student 6, a grade 2 student, with test accommodations already included in his or her IEP presumably for next year’s test administration; this was because grade 3 was the first grade of the state’s large-scale assessment. One student took the alternate assessment in lieu of the regular state test. All three students who used accommodations on the state test took the test in a small group setting with extra time available. Student 6 also was allotted breaks during the test administration, and student 8 also had the test directions read aloud to him/her. All students’ language abilities had been assessed in the past two school years, and resulting English language levels ranged from 2 to 4. Eight of the nine students received some type of special education services; service information was not available for student 9. Six of those eight students received both in-class and pull out special education services and one of those students (student 8) also received speech and language therapy. Student 5 was placed in a self-contained special education setting, and student 6 received only pull out special education services. Five of the nine students received second language services in both special and general education. Student 5 received these services in his or her self-contained special education setting. Second language services were not known for three of the students in School 1. Table 2 displays student assessment and services information for these students. Table 2. Student Assessment and Services Information for School 1
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||