NCATE/BOT Institutional Report

Previous | Index | Next

Standard 2: Assessment System and Unit Evaluation

The College of Education and Human Development (CEHD) uses systematic data from multiple sources and informants to monitor and improve unit operations, program effectiveness, and candidate performance. The college also evaluates its data systems to ensure that data are accurate and useful for program decision making. The college improves its data systems through the use of technology and the introduction of new systems of accountability.

Element 1: Assessment System

The unit assessment system is a comprehensive system for monitoring and assessing college information, unit operations, program effectiveness, and candidate performance. The system has three major components: unit data, department data, and the candidate assessment systems for initial and advanced programs.

Unit data
Unit data is collected and monitored by central college offices — the dean’s office and the office of Student and Professional Services (SPS). These data include information in four categories: program information, personnel information, information about college resources, and information about graduates and completers. Program information consists of several types of data. Among them is information about programs that is monitored, updated, published, and distributed to current and prospective students. SPS is responsible for managing information about program requirements as well as application and admission procedures for all undergraduate, initial licensure, and M.Ed. professional studies programs in the college. Most of this information is also available on the Web, either at the CEHD site or the SPS site. Similar program information for advanced programs (except for M.Ed. professional studies programs, as noted above) is guided by department and graduate school policy and maintained through the college communications office. The college also maintains information about how initial licensure and endorsement programs governed by the Minnesota Board of Teaching (BOT) meet Minnesota standards. Grids documenting the alignment of courses and field experiences with the Minnesota Standards of Effective Practice for Teachers (MNSEPT) are maintained by SPS and are available at the intranet site for review by program areas. The administrative licensure programs keep data on how their programs align with the standards of the Minnesota Board of School Administrators, while the Department of Educational Psychology keeps information about how the school psychology and school counseling programs meet state standards. Course syllabi for all initial and advanced programs are maintained at the department level.

Personnel information is maintained by the dean’s office and departments. It includes demographic data about all college faculty and staff, as well as performance reviews and data from course evaluations. Also, the dean’s office reviews the vitae and accomplishment reports of faculty and Professional and Administrative (P & A) staff annually. These data are used in promotion and tenure decisions. Additionally, SPS collects and maintains information about the performance of student teaching supervisors through the use of two assessments (see Standard 3). Data on unit resources is also maintained by the dean’s office. These data include information about budget, technology, and facilities.

Data on graduates and completers is collected and/or maintained by SPS. A rich source of information about program completers and feedback about initial and advanced programs comes through the administration of follow-up surveys. These surveys are administered by SPS and reports are made available through the college intranet and the Teacher Education Data System (TEDS). Table 2.1 summarizes these surveys.

Another source of data about program completers comes through the unit’s advisory boards. The Council on Teacher Education (CTE) has a separate advisory board that includes representatives from college administration, the state legislature, other institutions of higher education, the state teacher association, local school districts, college alumni, and the student body. This board meets twice a year and the agenda always includes an opportunity for members to provide feedback about the performance of graduates and program completers in the P-12 schools. The administrative licensure programs have several advisory boards made up primarily by practicing or recently retired school and district administrators. The college has initiated a new college-wide advisory board to provide guidance to the college at large. It met for the first time during Spring 2005. Individual departments and/or programs also have advisory boards or panels that meet intermittently to provide feedback and oversight to program faculty.

Table 2.1 Follow-up surveys

*Note: The How Their Careers Began – Non-licensure version is administered to graduates of undergraduate programs that do not prepare individuals for P-12 settings.

**Note: The Graduate Survey was conducted for the first time in 1999. A new version of the survey was conducted in 2004-05 and will continue to be administered as a regular part of the assessment system.

Department data
Department data is unique to specific program areas and is collected by faculty and staff within each department and/or program. Beginning in late 2004, the unit has begun a practice of periodically surveying programs about these data through the use of a tool called the framework for program evidence. Though this framework was developed as part of the preparation for the NCATE/BOT review of Fall 2005, it will be a useful tool for tracking program data on a regular basis in the future. Included in these data are information about program structure, candidate assessments (other than those conducted by the unit), field experiences, program improvement efforts, and implementation of the themes of the conceptual framework.

Candidate assessment system for initial programs
Candidates for initial teacher licensure are assessed on a series of benchmarks throughout the program. These benchmarks are used across all licensure programs, both undergraduate and post-baccalaureate. Candidates are assessed at admission, during foundations coursework, during methods coursework, during practicum and student teaching experiences, and at program completion. Post-completion, a sample of program completers are evaluated through the Employer Survey referenced above. The candidate assessment system for initial programs is summarized in the teacher education assessment system graphic.

Admission
At admission, each candidate is assessed on the following criteria:

1. BA/S or appropriate coursework: The candidate must hold a baccalaureate degree (for post-baccalaureate licensure programs) or have taken certain prerequisite courses (for undergraduate licensure programs).
2. GPA: The candidate must have a grade point average of 2.8 or higher.
3. Content coursework: The candidate must have taken certain content prerequisite courses that demonstrate attainment of the BOT content standards.
4. Classroom experience: The candidate must have at least 100 hours of volunteer or paid classroom experience. This criterion addresses the conceptual framework theme of lifelong learning and professional development.
5. Diversity experience: The candidate must have extensive experience with diverse populations. This criterion addresses the conceptual framework theme of diversity.
6. Essay: The candidate is evaluated on the quality of a reflective essay or goal statement. This criterion addresses the reflection component of the conceptual framework theme of inquiry, research, and reflection.

These criteria were established in the mid-1990’s by SPS in cooperation with the CTE and program area faculty. The candidate is assessed on all of these elements by a professional adviser in SPS through the use of an admissions scoring grid. After all applicants have been scored, the professional adviser meets with program area faculty to review the applicants and select the candidates who will become members of the program area cohort. Given the completion rate of these admitted candidates and their success in schools, it is clear that the selection process yields strong candidates for the program.

In addition, before beginning coursework in the program, the candidate must have taken the Praxis I exam of basic skills. This is not an element of the admission decision; however, the candidate must have taken the exam once before beginning coursework. It is included in the decision to recommend for licensure.

Foundations coursework
During foundations coursework, candidates are assessed on their knowledge and skills in meeting the Minnesota Standards of Effective Practice for Teachers (MNSEPT) as evidenced by course grades. Grids outline where standards are placed in all courses in the program, and course syllabi articulate how these standards are addressed in the course. During their foundations coursework, candidates are encouraged to begin documenting their progress in meeting the MNSEPT through the use of eFolio Minnesota, an online electronic portfolio system that is free for use by all citizens of the state of Minnesota. Candidates are encouraged to use an electronic portfolio to reflect on their work in the program, in accordance to the first central theme of the conceptual framework (inquiry, research and reflection). In addition, candidates use the electronic portfolio as an initial tool in their own professional development, reflecting the third central theme of the conceptual framework (lifelong learning and professional development). Faculty and staff in the college have worked with the eFolio Minnesota representatives to create a University of Minnesota teacher education template that incorporates the MNSEPT. Each spring during orientation, candidates are given an introduction to eFolio Minnesota. In addition, all candidates are taught to use the system in EDHD 5007 Technology for Teaching and Learning. Also, an online tutorial has been developed and is available to all candidates. It is important to note that at this time the use of eFolio Minnesota is recommended to candidates and program faculty, but is not required. Faculty in all initial licensure programs require the use of portfolios, while faculty in over half of these programs actively encourage the use of electronic portfolios, and all but one of the programs employing electronic portfolios uses eFolio Minnesota. In order to increase the number of programs using eFolio Minnesota as well as to ensure best practice in the use of this electronic portfolio system, CEHD staff in SPS and CI developed a project for training for both candidates and faculty in the effective use of eFolio Minnesota. College funds for completing this project have been secured.

Dispositions Assessment
The Dispositions Assessment was developed over a period of two years by the CTE sub-committee on assessment, beginning in Spring 2002. Throughout the process, feedback was sought from program faculty as well as the CTE Advisory Board. The CTE approved the final form and assessed candidates using the instrument for the first time in Spring 2004. The Dispositions Assessment incorporates standards drawn from the MNSEPT that relate to behaviors indicative of positive dispositions toward learning and teaching. Candidates are introduced to the Dispositions Assessment at orientation for new candidates. A key goal of the adoption of the Dispositions Assessment was noting and documenting concerns about a candidate’s dispositions before student teaching. In the first two years of implementation, it has been determined that low scores on the Dispositions Assessment are predictive of difficulty in the program. The Dispositions Assessment is aligned to the MNSEPT and the central themes of the conceptual framework. The Dispositions Assessment is part of the Teacher Education Data System (TEDS) (see explanation below under element 2). The Dispositions Assessment has two forms, the comment form and the evaluation form. The comment form is intended for use by foundations instructors and other personnel who interact with candidates but do not have responsibility for formally evaluating them on the Dispositions Assessment. The evaluation form is used by program area personnel to make formal evaluations of candidates’ dispositions. (See methods coursework section below.) Foundations instructors are encouraged to use the Dispositions Assessment comment form to document candidates who struggle in certain dispositional areas or those who stand out as exemplary in these areas.

Methods coursework
During methods coursework, candidates continue to be assessed on their knowledge and skills in meeting the MNSEPT as evidenced by course grades (Grids show this alignment.) Specifics about how the MNSEPT are addressed in coursework are articulated in course syllabi. It is during methods coursework that candidates work closely with program area faculty who bear the most responsibility for determining whether they are suited for the classroom. As such, it is the program area faculty who are responsible for assessing each candidate on the Dispositions Assessment evaluation form. The CTE recommends that candidates be assessed on their dispositions 10 to 12 weeks before the final student teaching placement. A second Dispositions Assessment is required for only those candidates who scored low on one or more items, scored “N” (no opportunity to observe) on one or more items, or who displayed new problems related to dispositional issues after the first assessment had been completed. Individual program areas are responsible for determining procedures for conducting the Dispositions Assessment within their programs. The Dispositions Assessment does not determine whether a candidate is allowed to progress through the program. However, if a candidate receives low marks on the assessment both times it is administered, a case conference can be triggered.

The case conference
Candidates’ progress during the program is assessed on an on-going basis by faculty. Problems that arise during methods coursework and continuing throughout the program are addressed through program-specific processes. However, if that is not effective, these concerns may be addressed through the use of a case conference. Case conferences are used for making decisions about entry into field experiences, removal or withdrawal from field experiences, reassignment to a new field experience, needed developmental activities, or terminating a licensure program. The case conference is usually attended by the college or school personnel who have raised the concern, the candidate’s faculty adviser, and representatives from SPS. Case conferences are documented and this information becomes part of the candidate’s record. For more information, see Standard 3. Historically, the case conference has been employed most often during student teaching. One goal of the Dispositions Assessment is to identify dispositional issues earlier in the program, allowing for early intervention and reducing the number of case conferences during student teaching.

Practicum/student teaching
The Formative Assessment of Practicum and Student Teaching Experiences was developed by the CTE clinical experiences sub-committee and approved by the CTE during 1999-2000. All elements of this assessment are referenced to the MNSEPT. During pre-student teaching practicum experiences, some program areas employ this assessment to monitor candidates’ progress. Other programs use components or sections of the assessment, while others use program-specific assessments. A few programs use this assessment throughout practicum and student teaching experiences. This assessment is not a required component of the candidate assessment system. See Standard 3 for more information.

Program areas and their departments are responsible for determining whether a candidate is ready for placement in student teaching. Procedures vary by program area and department. (See Standard 3.) During student teaching, the University student teaching supervisor bears the primary responsibility for assessing the candidate.

Evaluation of Student Teaching
Midway through the student teaching term, all candidates are assessed through an instrument called the Evaluation of Student Teaching. This is the same form that is used at the end of the experience as a final assessment. The use of the form midway through the experience is intended to be formative and provides candidates with the opportunity to improve their practice during the remaining weeks of the term. The current form of the Evaluation of Student Teaching has been used since 1999-2000 and is aligned to the MNSEPT and to the central themes of the conceptual framework. The final Evaluation of Student Teaching is considered a high-stakes assessment; a candidate must pass this evaluation in order to pass student teaching.

Standards Rubric
The Standards Rubric is used by program areas to assess candidates on their attainment of the MNSEPT over the course of the licensure program. This assessment was developed by the CTE sub-committee on assessment during summer 2003. It was adopted by the CTE in Fall 2003 and used for the first time during Spring 2004. In developing the Standards Rubric, college faculty and staff referenced Charlotte Danielson’s “Framework for Teaching” (Danielson, 1996) as well as the framework for evaluating teachers that is used by the Minneapolis Public Schools during the induction period (Minneapolis Federation of Teachers & Minneapolis Public Schools, 2003, pp. 56-72). Candidates are introduced to the Standards Rubric at orientation for new candidates. It is up to program areas to determine how and when to employ this assessment within the course of the program. Most programs assess candidates on this instrument upon completion of student teaching. Program areas are encouraged to review candidate portfolios when completing the Standards Rubric. The Standards Rubric aligns with the central themes of the conceptual framework. At this time, the Standards Rubric is not considered a high-stakes assessment; the CTE and its sub-committee on assessment are continuing to review data from the assessment and to consult with program faculty to determine whether it would be appropriate to make passing scores on the Standards Rubric a condition of licensure recommendation.

Follow-up surveys
As discussed earlier, the unit conducts two surveys of initial licensure program completers after they are in the field. The How Their Careers Began Survey has been conducted annually since 1975 and is administered during the first year after program completion. The primary goal of the survey is to determine whether completers have found employment in teaching and to learn about the characteristics of their teaching positions. Also, the survey queries respondents about college services. Additionally, there is a short bank of questions on the survey in which completers rate their preparation for teaching. Two open-ended questions seek further information about their experiences in the program. Conducted annually since 1982, the Second-Year Professional Survey is administered during the second year after program completion. Through this survey, completers update the college on their employment status, rate their preparation for teaching on a variety of factors, and provide information about their professional activities. In addition, respondents comment on the program through two open-ended questions. Both of these surveys align with the MNSEPT and the central themes of the conceptual framework.

Every three years, the unit conducts the Employer Survey. This survey is mailed to a random sample of Minnesota principals who employ initial licensure program completers during their third or fourth year of teaching. This survey has been conducted for more than 15 years. The items on the questionnaire are drawn directly from the MNSEPT and correspond to similar items on the Second-Year Professional Survey. The Employer Survey was reviewed and approved in June 2004 by the University’s Institutional Review Board to ensure that the survey complies with guidelines governing research on human subjects. The Employer Survey aligns with the MNSEPT and the central themes of the conceptual framework.

Candidate assessment system for advanced programs
The advanced programs hold the following transition points in common: application/admission, progress monitoring throughout the program, and completion. These transition points are addressed in ways that are specific to individual program goals. The following programs/program groups will be addressed in this section: M.Ed. professional studies programs, school psychology, school counseling, administrative licensure programs, and educational administration degree programs.

M.Ed. professional studies programs
At admission, applicants are assessed on the following criteria. Applicants must: possess an undergraduate degree from an accredited institution, demonstrate a strong academic record, demonstrate relevant work experience, demonstrate solid writing skills, provide a written statement, provide a resume, and possess a GPA of 2.80 or above. (There is one exception to this: The agricultural education program accepts students with a GPA of 2.50 or above.) Admission decisions are made by the M.Ed. professional studies advisers in SPS in consultation with program faculty.

After being admitted to the program, each candidate is assigned a faculty adviser who helps to monitor the candidate’s progress throughout the program. In addition, SPS professional studies advisers track candidate progress by monitoring completion rates and corresponding with students who have let time lapse between courses. Candidates have up to seven years to complete the M.Ed.

In most programs, a culminating project is required. The purpose of this project for candidates is to draw on their graduate studies and apply them to professional practice. To graduate, candidates must complete all program requirements and submit the final faculty-approved program plan. For example, the culminating project for the M.Ed. in teacher leadership requires candidates to identify a problem of practice, review related literature, then create a project to address the problem under the individual guidance of a faculty adviser. The projects range widely in focus and format. Some candidates conduct a classroom-based study of different instructional methodologies. Others might design, develop, and implement a staff development program. The overarching purpose of the project in teacher leadership is for candidates to lead some type of development or evaluation initiative in the particular context of practice.

Dispositions
The unit defines dispositions for M.Ed. professional studies programs as dispositional behaviors and practices related to the three central themes of the conceptual framework: inquiry, research, and reflection; diversity; and lifelong learning and professional development. Candidates in the college are assessed on their dispositions in a variety of ways within coursework. During the 2004-05 academic year, the Advanced Programs Representative Committee developed a dispositions survey for M.Ed. faculty advisers. Through the survey, faculty advisers had an opportunity to rate recent M.Ed. graduates on how often they displayed key dispositions related to the conceptual framework themes. In addition to rating the candidates on their dispositions, faculty advisers also provided information about their knowledge of the candidates’ participation in professional development, leadership, and advocacy. Finally, faculty advisers answered an open-ended question about the steps they take to help a candidate if they recognize that the candidate is lacking in one or more dispositional areas. The survey was conducted for the first time in Spring 2005. See Standard 1 for information on results from this survey. Dispositions are also addressed on the Graduate Survey.

Administrative licensure programs
The first major unit assessment in the administrative licensure program is the preassessment. This is a specific requirement in Minnesota Rule 3512. At the University this is a process as much as it is an end product. The process begins with the first course, EdPA 5385 Licensure Seminar, and its product is used to help guide the candidate’s learning plan. This plan, called the plan of action, becomes content for the resume and vita, helps to identify products for the eFolio Minnesota, and becomes the summaries of evidence for each competency in the eFolio Minnesota to be used in the exit panel assessment. Examples of preassessments: K-12 Principal (1), K-12 Principal (2), K-12 Principal (3), Superintendent (1), and Director of Special Education.

The preassessment is then submitted to the candidate’s faculty adviser who reviews it and makes recommendations for changes. A copy of the accepted plan is kept on file for each candidate. The candidate must complete this requirement prior to receiving permission to enroll in the field experience. The preassessment begins the development of the integrated thought process in which the candidate learns to analyze past experiences, looking for how various competencies work together to form a whole. In performing these steps, the learner develops a deeper understanding of the meaning of each competency and sub-competency.

The primary assessment for evaluating candidates’ attainment of the competencies is an electronic portfolio; the program uses eFolio Minnesota. The portfolio contains exhibits that collectively demonstrate all of the competencies and subcompetencies required in rule. See also creating the portfolio. The oral presentation of the portfolio is a key element in final evaluation of the candidate.

Approximately 80% of the candidates pass their review panel the first time. The others receive a Professional Development Plan (PDP), which identifies only those competencies that the candidate did not pass at entry level. The PDP identifies specific competencies and learning tasks to assist the candidate to gain the necessary knowledge, skills, and abilities to perform at a minimum of entry-level. Upon completion of the PDP the candidate is required to either meet with his or her adviser to verify competence or meet with a second or third full panel. This is the decision of the first full panel and depends on the extent, breath, and depth of deficiency. Examples of PDP’s: K-12 Principal, Superintendent, and Director of Special Education.

The panel members include three or more individuals, with at least one representing the University and serving as the panel facilitator, at least one other member representing practicing administrators in the license area under consideration, and the third member representing either the University or the field. Sample evaluation forms: K-12 Principal Licensure Portfolio and Evaluation Form; Superintendent Licensure Portfolio and Evaluation Form; Director of Special Education Licensure Portfolio and Evaluation Form

Prior to the panel review each candidate is required to participate in a pre-panel meeting with one of the advisers. These conferences have multiple purposes, including review of the candidate’s portfolio to determine if it is ready to be presented, coaching on techniques for the oral presentation, review of the candidate’s folder to determine that all other requirements are met, including completion of credits, and completion of the candidate’s pre-panel checklist. If it is determined that the candidate is not ready, the panel date is rescheduled.

Once the candidate has passed the panel, completed all credit requirements, and submitted the required paperwork, a letter is sent to the state licensure office along with a completed packet of supporting paperwork. The letter is the University’s attestation that all requirements are met and that the individual is recommended for an administrative license.

Educational administration degree programs
Candidates applying to the degree programs in educational administration (M.A., Ed.D., Ph.D.), must: possess an undergraduate GPA of 3.0 or above and a 3.5 graduate GPA; possess GRE scores over 500 on Verbal, 500 on Quantitative, and 4.5 on Analytical Writing; score 550 or above on the TOEFL of 6.5 or above on the IELTS (international students only); and submit a resume and two references.

Program handbooks provide comprehensive information about the master’s culminating project. In brief, the M.A. culminating project is a Plan B paper. Candidates are required to conduct and then write an in-depth, synthesized review of literature on a topic relevant to their program objectives and individual interests.

Candidates in the degree programs must meet admission requirements in order to eventually do field study research for their dissertations. They must have strong recommendations (many of which come from practitioners in P-12 school systems), personal statements that reflect significant commitment to education, and a robust record of previous academic achievement and talent. Candidates also must pass rigorous oral and written preliminary exams over their research topics and designs before approaching their P-12 field sites for data collection. Candidates must also get permission from districts to do their field studies. Finally, candidates must submit their research plans and protocols for review and approval by the University’s Institutional Review Board. In sum, candidates are not allowed to do research fieldwork or studies until they are approved at several levels of evaluation and assessment.

School counseling
At admission, applicants are assessed on the following criteria. Applicants must: possess a GPA of 3.0; submit GRE scores; have relevant employment and life experiences; provide a written personal statement; provide letters of recommendation; and international students must score at least 550 on the TOEFL, 80 on the MELAB, or 6.5 on the IELTS. Students who already possess a master’s degree can apply for the licensure-only program. Each application to the licensure-only program is considered separately and the program is handled on a customized basis and must meet the same criteria as those used in the master’s program.

Candidate progress throughout the program is tracked through the use of the Status in M.A. Graduate Training Questionnaire. On this form, candidates report their academic progress, academic plans, and professional goals. The information gathered through this form is part of the program’s annual review of students. The faculty review these forms as well as candidate transcripts and information from faculty advisers to determine how well candidates in the program are progressing. After the review, each candidate receives a letter informing them of whether or not they are making satisfactory progress.

Practicum candidates are required to periodically submit examples of their counseling skills for evaluation. These examples are submitted through an audio or visual taping session conducted at the practicum site. With a taped counseling session the supervisor may verify evaluations and conclusions about the candidate. Also, the clinical faculty can check on skills the candidate is using during the practicum experience. It is important for students to be able to ascertain strengths and weaknesses in their own counseling repertoire of skills.

School psychology
Applicants to the school psychology program are scored on a composite of: GPA (3.0 minimum), GRE scores, letters of recommendation, applicant statement/goals, and interview. Applicants must also demonstrate adequate undergraduate preparation in psychology and education.

Throughout the program, candidates are monitored on their academic progress, progress toward meeting program requirements, field placements, research and teaching experiences, employment, and professional activities. These factors are monitored at the program level with involvement from the faculty adviser. All candidates receive feedback from their advisers on a feedback form. Should there be a concern about the candidate’s performance, an information sharing/problem solving conference is available when difficulties arise. This conference is attended by the candidate, the faculty member(s) involved in the situation, and/or the faculty adviser. At the conference, a plan of action is determined and the candidate must sign a form accepting the plan of action.

During field experiences, candidates are assessed through the use of the Evaluation of Trainee form. This form was developed in 1998 and is used for both the Ed.S. and Ph.D. candidates. Through the use of this form, candidates are assessed on a variety of factors predictive of success in the field. These factors align with the standards of the National Association of School Psychologists (NASP). The form is completed by field-based supervisors who are school psychologists in the schools. In addition, a Teacher Feedback Form or Mentor Feedback Form is used to assess the candidate’s practice in the school setting. This form is completed by classroom teachers supervising the candidates. These feedback forms have been in use since 1993.

To move into the final phase of the Ed.S. or Ph.D. program, candidates must meet the requirements of the graduate school. In addition, candidates must present a satisfactory portfolio, personal competency statement, and personal specialization paper. Ed.S. candidates complete all requirements listed on the Specialist Certificate Special Field Prelim Requirements Form. The Ph.D. candidates complete all the requirements listed on the Ph.D. Special Field Prelim Requirements Form.

Follow-up survey
The Graduate Survey, discussed above, provides a rich source of data about graduates and completers of advanced programs. Survey respondents evaluate the program and their preparation for their roles in P-12 schools. In addition, they provide information about their professional activities. This survey was originally administered in 1999. It was rewritten and sent to a broader audience in 2004-05. The unit plans to conduct this survey bi-annually during the first and second years after program completion.

Element 2: Data Collection, Analysis, and Evaluation

Initial programs
The Teacher Education Data System (TEDS) was launched during the 2003-04 academic year. This is an online system for viewing and submitting initial licensure candidate and program data. This system brings together several previously existing databases that track candidates and program data, so that for the first time key data elements are accessible via the same system. TEDS was launched at the same time as the two new assessments adopted in 2003-04, the Dispositions Assessment and the Standards Rubric. The following table summarizes the data that are available in the system, the personnel who have responsibility for submitting the data, and personnel who have access to the data.

Table 2.2 Data available in the Teacher Education Data System (TEDS)

*These data are viewable only by individuals who have been approved for access to UM Reports, a secure online data system housing private student data.

TEDS is a secure system. All TEDS users must log in using central University authentication. Before being granted access to TEDS, users must agree to security and data privacy policies. These user agreements are repeated annually. TEDS developers met with University legal staff from the Office of General Counsel (OGC) before launching the system to make certain that proper security and data privacy guidelines were addressed. TEDS policies and procedures were reviewed by an OGC attorney.

As indicated above, much of the assessment data for initial programs is available through TEDS. In addition, complete survey reports are posted on the SPS intranet site. The CTE sub-committee on assessment and the coordinator for program review make regular reports of candidate and program data to the CTE, to SPS, and to the CTE advisory board.

Advanced Programs
The Advanced Programs Representative Committee (established in 2003) is a body of program representatives from advanced programs across the CEHD that is responsible in part for reviewing assessment data, planning new assessments, and bringing unity to these disparate programs. Major accomplishments related to assessment include articulating current data practices to the unit, developing a graduate survey, and developing a plan for assessing candidate dispositions. Additionally, assessment data at the advanced level are summarized and analyzed at the department level. Program faculty are accountable to department chairs in terms of providing assessment data on programs, processes, and candidate progress.

Unit level assessments
The unit prepares an annual report of student data each year, summarizing data on admission, enrollment, and completion/graduation for all programs within the college. Additionally, annual federal Title II reports are an important data source for unit personnel. These reports allow the college to benchmark enrollment and state test passing rates against earlier years and against statewide performance of all institutions of higher education. Annual AACTE and NCATE reports provide detailed information about the quality of candidates, and the key structures in the programs such as candidate progress and clinical experiences.

For initial programs, TEDS is an important source of information for faculty, professional advisers, and college administrators. This Web-based system brings together separate systems that track candidate data. Faculty and staff who work with teacher education programs in the CEHD use TEDS to submit unit-wide assessments — such as the Dispositions Assessment and the Final Evaluation of Student Teaching — and to view assessments submitted by others. The tool is used to help faculty communicate with each other about a candidate’s progress. Multiple faculty can provide input on a candidate’s assessment. The faculty adviser assigned to a candidate is given a higher level of access to that candidate’s data, thereby allowing him or her to view additional data such as admission scores and Praxis exam data.

In addition, TEDS is a source of aggregated data from candidate assessments (including admission scores, Praxis exam data, and unit-wide performance assessments). Also, TEDS houses survey data; these data are available in aggregated and disaggregated form.

Administrators, faculty, and educational policy committees are all involved in reviewing assessment data and making program recommendations. Primary individuals and groups responsible for data collection, maintenance, review, and analysis in professional education programs are the associate dean for undergraduate and professional programs, the coordinator of program review (SPS), CTE, CTE sub-committee on assessment, CTE clinical experiences sub-committee, Advanced Programs Representative Committee, Computer Information and Technical Services (CITS – dean’s office), SPS technical services, academic departments, and individual program faculty.

Also, the unit maintains a record of formal candidate complaints and documentation of their resolution.

• SPS designates “do not throw” files. This is a set of student files in which grievances or other difficulties are documented. These files are stored separately and are not purged at regular file clearing intervals.
• The Student Scholastic Standing Committee reviews student petitions, appeals, and academic progress. Minutes and records of their decisions are maintained.
• Departments maintain individual records of student requests, petitions, and the outcomes of grievances and case conferences.
• Student Dispute Resolution Center is a campus-wide resource to assist students in resolving disputes with faculty and grievances.
• There is an appeals process for admission and petition decisions; records are maintained in individual student files at SPS and, in the case of petitions, in a central notebook.
• Case conferences are recorded in student files. The clinical experiences coordinator makes an annual summary of the number of conferences and their outcomes.

Throughout both initial and advanced programs, assessments are regularly reviewed and evaluated to eliminate sources of bias and to ensure fairness, accuracy, and consistency in assessment procedures. Also, assessment results are analyzed in order to learn whether they are predictive of success in the program and how candidates are assessed across instruments on similar sets of knowledge and skills. In the initial programs, results from unit-wide assessments have been analyzed, with the following findings:

• Nearly all of the students who require case conferences to review their progress in the program scored low on the Dispositions Assessment.
• There is a positive correlation between a high score on the essay/goal statement completed by program applicants and a high score on the final Evaluation of Student Teaching.
• Students completing our programs possess strong content knowledge and instructional skills.
• Working with families and communities is an area of concern for many of our program completers. The unit has already begun to address this issue.

Element 3: Use of Data for Program Improvement

The unit uses data to drive decision-making and improvements throughout its program. Data-driven decision-making is evident in unit performance, departmental programs, and candidate performance.

Initial programs
The following actions are a result of data collection, analysis, and reporting:

• Evidence from the Second-Year Professional Survey and the Employer Survey indicate that completers’ skills profiles are generally strong, but relatively weaker as they come to meeting the needs of academically diverse learners, classroom management, and working with parents and families.
• In response, the second languages and cultures program developed a 1-credit workshop on meeting the needs of P-12 students learning English as a second language. This course, CI 5644 Working with Linguistically and Culturally Diverse Students in the Mainstream Classroom, was added to the following programs: art, elementary, math, communication arts/literature, science, and social studies. Candidates in other programs incorporate activities to adjust lessons to meet the needs of all students.
• To address classroom management concerns, the CTE reviewed which courses had those competencies embedded in them. Foundations faculty with course content related to classroom management made presentations to methods faculty at CTE meetings. Methods faculty consequently further aligned and coordinated their course content on classroom management.
• In response to concerns about communicating with families and communities, the CTE heard faculty presentations about field assignments used to help candidates get experience with families and the communities in their clinical settings.
• First-year and second-year follow-up survey data revealed lower ratings for advising services and assistance through the licensure process:
  • Several steps were taken to improve advising services including:
  • Improved consistency in SPS advising through the introduction of advising Web sites, adviser handbooks, and regular meetings between initial program advisers and advanced program advisers.
  • Revised advising model for the elementary education program by assigning the same SPS adviser to work with candidates completing the undergraduate elementary major (non-licensure) all the way through completion of licensure (post-baccalaureate licensure).
  • Improved use of the Web, especially to assist admitted candidates with initial planning and enrollment, and the initiation of the You’ve been accepted! Web site
  • Online sign-up for information sessions and fingerprinting to ease scheduling for prospective students and candidate completion.

Data-driven improvements are also evident for clinical experiences, including support for cooperating teachers, preparation of University supervisors, and information management. The college offers significant support for cooperating teachers. Attendance at cooperating teacher orientations was poor (ranging from 5 to 15 attendees in 4 offerings over two academic years), but teachers who attended spoke to the value of the sessions. In response to this, orientation materials were made available online. The student teaching handbook was posted to the Web in 2003, while a “frequently asked questions” page, tip sheets, and links to more information were added in 2004. Online asynchronous orientation modules, which provide renewal units, will be available in Fall 2005. This strategy has improved access. In February 2005, we averaged 6 Web site hits and 27 page views per day; in March 2005, we averaged 6 site hits and 18 page views per day.

The college provides substantive preparation for University supervisors. Analysis of student teaching case conference records revealed that new University supervisors felt unsure of their roles and responsibilities. An orientation for new University supervisors was introduced in 2002. An analysis of case conferences also revealed that University supervisors needed more preparation to solve problems in the field. In 2003, a workshop on field-based problem solving was added to the training sequence.

Discussions with University supervisors at these workshops revealed that many supervisors felt out of touch with their colleagues in other licensure areas, and with broader teacher education initiatives in the college. In 2004 the New Era workshop was added to the training sequence. New Era workshops present information about new developments and initiatives so the content will change each year. Additionally, a moderated electronic mailing list for all college supervisors launches in Fall 2005. This listserv will facilitate information sharing and professional interaction among the college’s 45 supervisors.

Advanced programs

The following are examples of program changes in advanced programs made as a result of data collection, analysis, and reporting.

• The administrative licensure programs responded to new guidelines around state competencies about the assessment of standards in a portfolio environment. In response new portfolio materials for candidates and improved pre-assessment processes have been developed.
• Music education M.Ed. candidates expressed a need for specific studies in their areas of concentration (i.e., choral, instrumental, orchestral, and general). The program developed separate pedagogical tracks for each specialization.
• The school psychology program, responding to feedback from their Evaluation of Practicum and Evaluation of Intern forms, better coordinated their intervention coursework and fieldwork. They also increased the number of school sites and time in schools that each candidate experiences. Responding to NASP and APA standards, the school psychology program changed the content emphasis of the practicum seminar from an issues focus to a focus on standards and ethics.
• The curriculum and instruction professional studies M.Ed. program hosted a meeting of program area representatives in which candidate work was discussed. Relative deficiencies in curriculum development skills were noted, and as a result the core class CI 5155 Contemporary Approaches to Curriculum, Instruction, and Assessment was redesigned to emphasize curriculum development in addition to the existing focus on instruction and assessment.

This is a sample of the data sources and changes that have occurred at the individual program level. These and other program changes can be seen at framework for program evidence along with the data-sources that led to these changes.

Assessment data sharing
The college shares assessment data for good decision-making and program improvement by all stakeholders. Assessment data are shared with faculty and other stakeholders through presentations and discussions at CTE, Advanced Programs Representative Committee, advisory boards, SPS all-staff meetings, TEDS, SPS intranet site, e-mails to faculty and staff, updates in “A Closer Look”, and the CEHD Web pages.

Previous | Index | Next