Judy Swanson: Welcome to a live chat with James Fox, Ph.D. on the topic of "Evaluating Your Team." Dr. Fox is Research Director for the Center for Early Childhood Learning and Development and Project Director of "Make a Difference" at East Tennessee State University.

Jim Fox: Hey, Jim Fox here...

Judy Swanson: Welcome, Jim!

Jim Fox: I've got a list of things but how shall we start this?

Judy Swanson: Let's hear what you're thinking. What is one of the first things we need to do to effectively evaluate our teams?

Jim Fox: Well, it seems to me that to talk about team evaluation you have to start at the end. The purpose of the teams is to come up with more effective function-based intervention. If we agree on this purpose, then we can start to back up toward the beginning of team evaluation. For teams to be able to do good FBA they have to be well trained. So we start team evaluation by assessing where they begin in their knowledge and where they end up at the end of training.

Now let's look at evaluating teams in training--we need a pre-test and at least a post-test of their knowledge of functional behavior assessment. We've also found it helpful to have some formative evaluation of training along the training sequence. To keep it simple but helpful, some type of brief objective assessment at the pre-test stage can be done. I think you folks at Minnesota have something like this and we are working on one also.

Formative assessment, we have a short little questionnaire of three questions we do each day of training. What was something helpful we did/you learned today? What is something we can do in training tomorrow to help you learn tomorrow? And, what is something you still don't understand? I'll stop here for questions and to let my staff get logged on.

Participant: One of our new teams is asking what they need to have in place to use their information for later research. What suggestions would you have for them?

Participant: Generally we have a time log for team members to enter the amount of time that they spend in team meetings, observations, interviews, etc. This is minimal information.

Jim Fox: Yeah, what I had been going over is what we do for evaluation during training. We hadn't gotten to team meetings yet. One other way we have to evaluate teams during training are case studies that they do throughout training.

Participant: Do you gather any information from the classroom staff that receive the technical assistance from the team?

Jim Fox: Yes, we get the recipient of technical assistance survey after we complete a case. Once training is complete, then we can evaluate teams in other ways as they begin meeting. During team meetings, we get the time budget questionnaire completed, each weekly meeting that Cindy mentioned. It shows how much time each team member spends each week on a number of FBA activities.

Also, when our project has directly done FBAs, then we have begun looking at the time from initial referral to first interview to case completion. I think that's another good way for a team to track its progress and ensure that it is keeping to a reasonable time line. I'm stopping here for comments or questions.

Participant: In developing a new team, have you had any thoughts you'd like to share?

Jim Fox: I think one of the things we've found most difficult is evaluating teams in terms of the final criterion, effectiveness of implementation, and intervention. It's not so much we don't know how to evaluate team effectiveness; it's the practical problems of doing follow-up observations.

Participant: Jim, any thoughts on evaluating the fidelity of implementing interventions?

Jim Fox: Yeah, keep it simple! I'd say have a very simple observation system for the critical target behaviors. Then, a simple checklist of the intervention components agreed upon to be implemented, and completing that checklist at the end of observation. So, to check fidelity, if the intervention were a point system, shortened activities or task, and tutorial instruction, list those on a short checklist and have whoever is observing note whether or not they saw this implemented during the observation of the child's behavior. Not the most detailed evaluation but certainly the checklist with some comments section would be sufficiently informative. Anybody have any other thoughts or experiences with any part of evaluation?

Participant: When release time is often an issue, it's hard to find the time to follow up like this, but really important to do to be certain the behavior support plan is working for the child.

Participant: Long-term, if you are trying to evaluate child change, we check with a phone call or visit to the teacher to check on improvements. We often have reports of improvement even if all recommendations were not implemented. Teacher behavior changed and attitudes toward the child and behaviors.

Jim Fox: Yeah, keeping the follow-up short, simple, but helpful is tricky.

Participant: We spend so much time evaluating, we sometimes feel our job is done once we have decided on an intervention.

Jim Fox: Yeah, that's why it's important to reflect back on the goal of FBA. It's not to do FBA but to develop more effective interventions.

Jim Fox: Yeah, I think there is kind of a positive expectancy effect with what we do sometimes. For some classrooms, almost any change can produce some positive outcome for the kid or teacher. I think I can give an overall summary of what kinds of evaluation questions we should ask. What kind of FBA information do team members need and did we teach them that information? Once the team starts meeting, are they applying that information, are they spending their time appropriately, and proceeding through FBA at a reasonable pace? Once the FBA is done, did it produce an intervention(s) that addressed child behavior? Was the intervention implemented as agreed upon, and did the recipient(s) of this assistance find it helpful.?

Jim Fox: I see we've come up on the hour. If anyone would like copies of evaluation forms or procedures we use or that we are developing, we can e-mail them and/or have them posted to this website, if that works for the University of Minnesota. Let me know. Thanks.

Judy Swanson: Thank you for joining us today. A special thanks to Dr. Fox. Yes, I'm sure we can arrange for the forms to be available on the site. This is our last scheduled live chat until the fall. We welcome your suggestions for future chat topics. Please e-mail me with your ideas. Also, let me know the best time of day for us to schedule the chats.

Participant: It was great talking to you.

Joe Reichle, Ph.D., Co-Principal Investigators
LeAnne Johnson, Project Coordinator
Phone: 612-625-3457 E-mail: watso012@umn.edu

[Home] | [Personnel] | [Presentations] | [Strategies] | [Case Studies] | [Discussion] | [Links] | [Contact Us]