WHO WE ARE    |   PUBLICATIONS   |  PRESENTATIONS   |  PROJECTS   |   RELATED SITES   |   STAFF   |   SITE MAP   |   SEARCH   |   WHAT'S NEW   |   HOME 

Relationships Between a Statewide Language Proficiency Test and Academic Achievement Assessments

LEP Projects Report 4

Published by the National Center on Educational Outcomes

Prepared by Kentaro Kato, Debra Albus, Kristin Liu, Kamil Guven, and Martha Thurlow

August 2004

Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:

Kato, K., Albus, D., Liu, K., Guven, K., & Thurlow, M. (2004). Relationships between a statewide language proficiency test and academic achievement assessments (LEP Projects Report 4). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/LEP4.html

Executive Summary

Minnesota is one of many states that began development of an English proficiency test before federal requirements were in place to do so. It had decided to put into place a test that would provide the state with a better and more uniform gauge of how its population of English language learners (ELLs) was doing in their acquisition of academic English language skills. Minnesota chose to adapt its test, the Test of Emerging Academic English (TEAE), from the Illinois Measure of Academic Growth in English (IMAGE). The TEAE is designed to gauge the growth of emerging academic English language skills across all grades, including three forms spanning grades 3-4, 5-6, and 7-8. The 7-8 form is also designed for use with students above grades 7-8.

This report focused on state ELL performance on the TEAE, in comparison to ELL and fluent English student performance on Minnesota’s Comprehensive Assessment (MCA) in reading in 3rd and 5th grade, and Minnesota’s Basic Skills Test (BST) in reading in 8th grade. The TEAE is designed to measure the basic English proficiency required for pursuing higher-level academic achievement, while the MCA is designed to measure academic achievement toward the state standards. The Basic Skills Test in reading measures the basic skills needed to be able to graduate. Across these comparisons, our guiding research questions were to find out what levels of the TEAE best predicts success on the MCA and BST, and whether the state decision to count as proficient those ELLs who achieve at level 4 on the TEAE has a sound base of support from an assessment perspective. Study 1 addresses the questions related to the TEAE and the MCAs. Study 2 addresses the same questions for the TEAE and the BST.

Key Findings:

Study 1: TEAE and the MCA

• ELLs in TEAE level 4 are likely to do as well as native English speakers on the MCA, recognizing that there is a range of performance among native speakers.

• Although the specific predictive relationship (i.e., what TEAE score corresponds to what MCA score) can differ, the positive relationship between students’ performance on the two tests is stable across years and grades.

• For students with TEAE scores below about 110, there is less ability to predict MCA scores.

• Most students in TEAE level 3 fall into MCA levels 2A, 2B, or 3 and therefore although it is likely that many within this group score as proficient (i.e., 2B or 3) others may not (2A).

Study 2: TEAE and the BST.

• TEAE scale scores had moderate predictive power for BST performance. However, the predictability is not as good as for the MCA.

• To predict that a student would be likely to pass the BST, he or she must score at least 260 (i.e., achieve level 3) on the TEAE.

In conclusion, there might be stronger relationships between the MCA and 3rd and 5th grade reading skills on the TEAE because the academic language skills measured on the TEAE fit those elementary grades better. Other factors besides potential discrepancies between secondary grade level skills and basic academic language skills may also account for differences in performance between the tests. These include differences in a learner’s age upon entering Minnesota schools, differences based on student familiarity or lack of familiarity with topical content and vocabulary for individual passages encountered on the tests, and teachers’ own anecdotal evidence which suggests that some students who take the TEAE do not take the test seriously. Any combination of these and other individual student factors could contribute to the TEAE not predicting success on the BST as well as on the MCA.

Overview

Minnesota is one of many states that began development of an English proficiency test before federal requirements were in place to do so. It had decided to put into place a test that would provide the state with a better and more uniform gauge of how its population of English language learners (ELLs) was doing in their acquisition of academic English language skills. Minnesota chose to adapt its test, the Test of Emerging Academic English (TEAE), from the Illinois Measure of Academic Growth in English (IMAGE). The TEAE, begun before Title III legislation required an annual growth measure for English proficiency under the No Child Left Behind Act of 2001, is now used to serve accountability purposes at federal and state levels, and is the official measure to provide on-going identification of English language learners in Minnesota for the purpose of state funding. This said, a student’s proficient scores on the TEAE reading and writing tests do not prohibit him or her from receiving on-going ESL/bilingual support as deemed feasible by local districts.

The TEAE is designed to gauge the growth of emerging academic English language skills across all grades, including three forms spanning grades 3-4, 5-6, and 7-8. The 7-8 form is also designed for use with students above grades 7-8. Gauging growth in academic English, and even defining it, is a challenge for language acquisition specialists and assessment specialists alike. The different viewpoints on what constitutes academic English (Bailey, Butler, LaFramenta, & Ong, 2004; Chamot & O’Malley, 1994; Cummins, 1979; Scarcella, 2003; Solomon & Rhodes, 1995; Stevens, Butler, & Castellon-Wellington, 2000), makes the design, implementation, and interpretation of such a proficiency test complex at best, especially when translating back the results into what academic language skills a student truly needs for success across content classrooms such as reading and mathematics.

This report focuses on state ELL performance on the TEAE, in comparison to ELL and fluent English student performance on Minnesota’s Comprehensive Assessment (MCA) in reading in 3rd and 5th grade, and Minnesota’s Basic Skills Test in reading in 8th grade (BST). The TEAE is designed to measure the basic English proficiency required for pursuing higher-level academic achievement, while the MCA is designed to measure academic achievement toward the state standards. The Basic Skills Test in reading measures the basic reading skills needed to be able to graduate. Across these comparisons, our guiding research questions are to find out what levels of the TEAE best predicts success on the MCA and BST, and whether the state decision to count as proficient those ELLs who achieve at level 4 on the TEAE has a sound base of support from an assessment perspective. Study 1 addresses the questions related to the TEAE and the MCAs, Study 2 addresses the same questions for the TEAE and the BST.

Study 1: TEAE and MCA

Method 

In Study 1, we use the Minnesota state test data of third and fifth graders in school year (SY) 2001-02 and 2002-03. Although the TEAE consists of reading and writing tests, we focus only on the reading test and its relationship with the MCA reading test. Hereafter, they are simply denoted by TEAE and MCA, respectively. The MCA data include test scores of all students who participated in the state assessment. The TEAE data consist of test scores of ELLs. The TEAE data originally contained 5,161 third graders and 4,688 fifth graders in SY 2001-02, and 5,123 third graders and 4,683 fifth graders in SY 2002-03. The MCA data originally contained 61,922 third graders and 64,408 fifth graders in SY 2001-02, and 60,018 third graders and 63,350 fifth graders in SY 2002-03. The data files for the same school year were merged using the student ID as the key variable. At this step, students with invalid or no student ID number were flagged so that they would not be used in the subsequent analyses. The merged data were then screened to exclude students who had any missing value on variables related to test scores (i.e., raw scores, subscale scores, and scaled scores; if any of these is missing, then other scores are not reliable even if they are recorded). Students who are recorded as “not tested” on MCA were also excluded. The resulting sample sizes are shown in the third column in Table 1.

Table 1. Descriptive Statistics for TEAE and MCA Data

Note. N is sample size, SD is standard deviation, and r is sample correlation between TEAE and MCA.

Next, we examined the relationship between the two tests. English proficiency as measured by the TEAE is considered to be prerequisite to minimal performance on the MCA. Thus, we expect that performance on the two tests is positively related, but detailed analysis will reveal more specifically the degree to which they are related. We analyzed the data in three ways based on how the results of these tests may impact practice.

The first analysis examines the relationship between the two tests at the scale score level. The scale scores of the TEAE and the MCA represent English proficiency and academic achievement toward the state standards, respectively. Every year performance on both tests is converted from raw scores so that they have similar distributions across years irrespective of changes in test items. Based on our research questions, we inspected scatter plots of the MCA and TEAE, and then applied regression analysis to examine the extent to which the MCA scale score is predicted by the TEAE scale score.

The second analysis focused on the relationship between the two tests by the proficiency or achievement level. The MCA has five achievement levels, I, IIa, IIb, III, and IV, based on cutoff points set on the scaled score. Students who are in level IIb or above are counted as “achieved” for accountability purposes in Minnesota. The TEAE has four levels to represent English language proficiency based on the scale score. On both the MCA and the TEAE, each level is associated with a specific description of progress toward the state standards (MCA) or English proficiency (TEAE), and thus gives a brief and clearer interpretation of a test result. Also, using such levels makes the results less sensitive to measurement errors on scale scores. Examining the relationship between the two tests by the proficiency or achievement level leads to relating a specific level of English proficiency to a specific achievement level.

The third analysis is motivated by the regulation that ELLs who have achieved the highest proficiency level (level 4 on reading and level 5 writing) on the TEAE are no longer eligible for funding for LEP programs because they are regarded as having English proficiency sufficient to access the academic content in the mainstream without further language support. If results of the TEAE reflect this reasoning, then the distribution of MCA scores of ELLs who are in the highest English proficiency level are comparable to those of students who are not ELLs. In other words, the means of the MCA score distributions of both groups of students should be almost the same and the ranges of the distributions should substantially overlap. Accordingly, the distribution of MCA scale scores for each of the TEAE proficiency levels will be compared with the distribution of native English speakers. Test scores of native English speakers were taken from the Minnesota state test data as well, and those data were screened in the same manner as for the TEAE.

 Results

Descriptive Statistics for the Entire Sample

Descriptive statistics by grade and year were shown in Table 1. Within each school year, fifth graders had higher mean scores on both the TEAE and MCA as expected. Fifth graders had larger variability on the MCA than third graders in both school years. Fifth graders had larger variability than third graders also on the TEAE in 2001-02, while there is little difference in 2002-03. Correlations between the TEAE and MCA are larger than .70 for all grades and years. This indicates an overall stable, positive relationship between the TEAE and MCA. Still, it is worthy of more detailed examination. 

Analysis of Scale Scores

Scatter plots. Scatter plots of MCA scale scores and TEAE scale scores by grade and year are shown in Figures 1 through 4. These plots consistently indicate that the majority of points are positively correlated. However, there is a group of points that do not follow that major pattern in the region where TEAE scale scores are less than a given point. For third graders in 2001-02, for example, data points with TEAE scores less than about 100 seem to have almost no correlation while the majority of data points are positively correlated. For these "irregular" points, MCA scores looked highly unpredictable based on TEAE scores. Thus, it is better to separate these points in order to investigate the relationship that applies to the majority of students in the data set. The question is, however, at what point we should separate regular and irregular cases; there is no indicator variable that separates these two types of points in the data files.

Figure 1. Scatter plot of MCA and TEAE scale scores (2001-02, Grade 3)

Figure 2. Scatter plot of MCA and TEAE scale scores (2002-03, Grade 3)

Figure 3. Scatter plot of MCA and TEAE scale scores (2001-02, Grade 5)

Figure 4. Scatter plot of MCA and TEAE scale scores (2002-03, Grade 5)

To estimate a cut off point for the scale scores for each grade and year, the following simple linear regression model is applied to the regular group of students (i.e., students with TEAE scores greater than the cutoff point) to assess the predictability of the TEAE on the MCA: 

MCA = (Intercept) + b₁ (TEAE) + e

Although there probably are multiple ways to estimate the cutoff point, a change point analysis is used for this purpose. It searches for the best cutoff point by fitting two different linear regression models for regular and irregular groups, respectively.

It should be noted that the TEAE scale scores show some discreteness in the score range above 300 (i.e., there are big jumps between two adjacent possible scale scores) in the score range above 300. This is more apparent for fifth graders, because more students marked scores close to the maximum possible scale score. This discreteness results from the scaling, which depends on the distribution of raw scores in each grade and year.

Estimation of Cutoff Scores

Estimated cutoff scores are shown in the third column in Table 2. The mean squared errors of MCA scores in the irregular group estimated by the change point analysis were 167.97 and 163.53 for grade 3 (2001-02 and 2002-03, respectively), and 132.15 and 201.24 for grade 5 (2001-02 and 2002-03, respectively). These are almost the same as the unconditional standard deviations listed in Table 1 except for fifth graders in 2001-02. Thus, we can conclude that MCA scores of students with TEAE scores less than the cutoff points are not well predicted by the TEAE. Although these cutoff points vary across years and grades, the unpredictability is likely to occur when the TEAE score is less than about 110.

Table 2. Estimates of cutoff scores and regression coefficients

 Note. Intercepts and slopes are for the “regular” group of students with TEAE scores greater than the cutoff point. N is the number of students included in the regular group, and R² is the squared multiple correlation.

Regression Analysis for the Regular Group

In the fourth through seventh columns in Table 2 are shown the number of students in the regular group, estimated intercept, slope, and R squared for the regular group of students (i.e., students with TEAE scores greater than the cutoff point). The slopes range from 3.31 to 4.29, and the corresponding R²s range from .54 to .58. These results indicate that more than 54% of variation of the MCA scale score can be accounted for by the TEAE scale score for the regular group of students. This is a strong positive relationship. The results also indicate, however, that slopes vary to some extent across years and grades. The estimated regression lines are plotted in Figure 5. As the slope estimates indicate, the lines are almost parallel except for grade 3 in 2001-02, where the regression line is slightly steeper than the others. Also, vertical locations of the lines vary in the 200 range for the MCA score scale. The lines for grade 3 are higher than those for grade 5 in Figure 5, but more longitudinal data would be required to infer systematic effects of grade levels on regression lines. Overall, although the specific predictive relationship (i.e., what TEAE score corresponds to what MCA score) can differ, the positive relationship between the two tests is stable across years and grades. Thus, we expect that increased English proficiency is associated with progress toward the state academic standards.

 Figure 5. Comparison of Estimated Regression Lines

Relationship by Proficiency or Achievement Level

Grade 3 TEAE Level and MCA Level Correspondence

Tables 3 and 4 show the number of third graders cross-classified by TEAE proficiency levels and MCA achievement levels in 2001-02 and 2002-03. Level 1 of the TEAE includes the “irregular” group of students found in the analysis of scale scores.

Both 2001-02 and 2002-03 results consistently indicated the following. First, students in TEAE level 1 are likely (about 80%) to be in level 1 on the MCA, and thus to be counted as "not proficient" for accountability purposes. This is a clear indication that basic English proficiency is a prerequisite to achieving higher-level academic reading skills. Second, students in TEAE level 4 are likely to achieve level 3 or 4 on MCA, and thus to be counted as proficient for accountability purposes (the result for 2001-02 may not be reliable due to the small sample size of 24 in TEAE level 4). Thus, proficient English learners can do well on the MCA. Finally, TEAE levels 2 and 3 seem to have no single corresponding level on the MCA. Most students in TEAE level 2 fall in MCA level 1, 2A, or possibly 2B, although they are unlikely to be proficient (2B) on the MCA. Also, most students in TEAE level 3 fall into MCA levels 2A, 2B, or 3. They are likely to be proficient on the MCA but there is still some possibility that they would not be proficient.

Although there is no clear one-to-one correspondence between the TEAE proficiency levels and the MCA achievement levels, ELLs who are in TEAE level 3 or 4 are likely to be proficient (i.e., scoring in level 2B or above) on the MCA.

Table 3. Correspondence between TEAE Proficiency Levels and MCA Achievement Levels (2001-02, Grade 3)