Developing an Observed Structured Teaching Exam at UCSF

Katherine Julian, M.D., Margo Vener, M.D., Maria Wamsley, M.D.
Fall 2003

Background Information
Traditionally, the quality of clinical teaching by faculty and residents is measured through written evaluations completed by learners. Most faculty and resident instructors teach in individual or small group settings; thus, it is often difficult to interpret student teaching evaluations. Individuals in a small group may vary and different learners have different needs. Student evaluations may often relate to a teachers’ charisma or communication style. Increasingly, Objective Structured Teaching Evaluations (OSTEs) are being employed as a method to allow instructors to practice teaching skills and to provide instructors with immediate feedback. OSTEs are a more rapid nd
objective method of assessing teaching skills and have been used to evaluate clinical teaching skills of medical faculty.

OSTEs have been described at many academic institutions but have never been used at UCSF. We propose to develop an Observed Structured Teaching Evaluation at UCSF. This OSTE pilot project would utilize medical student “actors” to simulate teaching situations. This project would target faculty and resident clinical teachers on the UCSF campus.

Goals
The goals of this project are:
1) To develop an Objective Structured Teaching Evaluation method at UCSF.
2) To provide key clinical faculty with the opportunity to participate in an OSTE and to provide faculty with feedback on their teaching methods.
3) To pilot the use of the OSTE with selected UCSF residents and provide these residents with feedback on their teaching methods.

Procedures
This project would target key clinical faculty and selected residents at UCSF. We plan to intensively train a group of medical students to serve as student “actors”. These student actors will be taught important teaching skills and will learn how to rate teachers on their teaching performances. OSTEs will be staffed by medical students in their fourth-year of medical school. UCSF School of Medicine is currently implementing a program to allow medical students to select an Area of Concentration during medical school so that students may develop expertise in a field of career interest. Our program will target
fourth-year medical students in the Medical Education Area of Concentration. Evidence demonstrates that medical students who serve as standardized patients in objective structured clinical examinations (OSCEs) may improve their own communication skills. Medical students participating in the OSTEs as standardized students may benefit by improving their own teaching skills. Medical students will receive 30 hours of training in the OSTE rating scales to ensure reliability and validity of the OSTE scores. This training will be conducted by Drs. Julian, Vener, and Wamsley as well as Bernie Miller--
a trainer who specializes in the training of Objective Standardized Clinical Examination participants. OSTEs will be held at the UCSF Clinical Skills Center. Medical students who elect to participate will learn this training as part of their medical education elective time and will receive six teaching sessions (taught by Drs. Julian, Vener, and Wamsley) on how to improve teaching skills.

We will develop 6 different OSTE scenarios. These scenarios will focus on: orienting a learner using goals/expectations, creating a positive learning climate, teaching a procedure, bedside teaching, small-group teaching, and feedback to a learner in difficulty. These scenarios are chosen because they are the most frequently encountered teaching skills needed by teaching instructors. We will train 4 students for each OSTE case so that there is sufficient “back-up” in case students cannot attend each OSTE session.

The OSTEs will be geared towards clinical faculty with significant teaching responsibilities at UCSF and a pilot group of multi-disciplinary residents. Clinical faculty will be recruited from faculty participating in the Longitudinal Clinical Experience (LCE), Foundations of Patient Care (FPC), and Family and Community Medicine (FCM) 110. LCE has approximately 100 clinical faculty in a variety of primary care and specialty settings. FPC has a similar number of outpatient preceptors for 1st and 2nd year students. FCM 110 utilizes approximately 40 clinical preceptors to teach 3rd and 4th year medical students. Our interdisciplinary faculty would collaborate with course directors to ensure that cases could be applicable across a variety of medical
specialties. OSTE cases will also be piloted with a select group of 12 multi disciplinary residents who are participating in the Resident Teaching Fellowship at UCSF for the 2004-2005 academic year.

For each OSTE session, there will be 6 OSTE stations. Each station will be 15 minutes in length and will assess a different teaching skill as noted above. Stations’ participants will be rated on a five-point Likert scale using a modified rating scale based on the SFDP-26.

For faculty, the OSTE sessions will be run twice during the academic year (October 2004 and March 2005). OSTE sessions will be held in the morning and afternoon. Thus, 12 faculty will be able to participate in October and 12 faculty will participate in March. The OSTE for residents will be held in February 2005 (at the conclusion of the Resident Teaching Fellowship); 12 residents will participate.

On the day of the OSTE, teachers will be given case scenarios in which they are instructed to engage in specific teaching tasks with students (such as orienting a new 3rd year student on a rotation). Teachers will rotate to through six exam stations (much like students do in the Clinical Practice Exams). Students will be trained on various aspects of teaching in their setting. Using a checklist of skills, students will rate each teacher after their encounter and also provide any additional written comments. All encounters
will be videotaped. At the end of the sessions, teachers will receive their videotape for review as well as the comments of the student raters. In addition, course directors will have discretion to offer individualized video review with any teachers having particular difficulties or concerns. Course directors could review the teacher’s goals, discuss strengths and challenges, and assist the teacher in developing an individualized teaching improvement plan. Finally, this is a potential opportunity for the Academy of Medical
Educators to develop a teaching clinic whereby faculty could elect to work with Academy members to further process any teaching skills learned during the OSTE sessions.

Plan for Measurement and Documentation of Project Efficacy and Outcome
Participating faculty and residents will complete pre- and post- OSTE Teaching Self- Efficacy scales as well as the Self-Assessment Inventory for Clinical and Classroom Teaching in Medicine.11 Participating faculty and residents will also complete overall evaluations on the OSTE experience (detailing the usefulness, quality of cases).

Plan for Continuation of Project at the End of the Funding Cycle
We anticipate that once the OSTE cases are developed, OSTE sessions could be offered yearly to interested faculty and residents participating in the Resident Teaching Fellowship. Participating medical students would have to be trained yearly to sustain this program. Bernie Miller has extensive experience in training actors for clinical cases but faculty effort would be needed for general supervision. To sustain this program, continued funding will be sought through faculty development grant applications as well
as from the Office of Medical Education.

For further information, please contact:
Katherine Julian, M.D.
Margo Vener, M.D.
Maria Wamsley, M.D.