2014 年 232 巻 3 号 p. 223-227
Problem-based learning (PBL) is a teaching and learning method designed to develop clinical reasoning skills. Tutor performance in PBL affects both the process and outcome of student learning. In this study, we investigated the factors that influence the evaluation by undergraduate students on the performance of tutors in medical education. From April 2009 to February 2010, 49 PBL sessions were conducted for 191 3rd- and 4th-year medical students at Saga Medical School in Japan. Twenty-nine 6th-year students and 205 faculty members tutored these sessions. After each session, students evaluated their tutor by a Likert scale. This evaluation score was dichotomized and used as the dependent variable. A multivariate logistic regression analysis was used to assess the contribution of student’s gender and year level (3rd or 4th), the tutor’s gender and background, and the quality of the case scenario to evaluation ratings. A total of 4,469 responses were analyzed. Male student and tutor background were associated with excellent tutor evaluation. Concerning the tutor background, compared with basic scientists, the 6th-year students and content-expert clinicians were positively associated with excellent tutor evaluations (ORs of 1.77 [95% CI: 1.15-2.72] and 1.47 [95% CI: 1.11-1.97]), while non-content-expert clinicians received negative evaluations (OR of 0.72 [95% CI: 0.55-0.95]). The quality of the case scenario was also associated with excellent tutor evaluation (odds ratio [OR] of 12.43 [95% CI: 10.28-15.03]). In conclusion, excellence of case scenarios, 6th-year student tutors, and content-expert clinicians show positive impact on tutor evaluation in a PBL curriculum.
Problem-based learning (PBL) is a case-based learning method designed to develop clinical reasoning skills including the acquisition, integration, and application of new knowledge (Newfeld and Barrows 1974; Norman and Schmidt 1982; Barrows 1994). In a PBL curriculum, the main role of the tutor is to facilitate the learning process by encouraging small-group discussions on case scenarios and giving formative feedback rather than teaching factual knowledge. It is widely accepted that the quality of tutoring by faculty members is one of the most important factors for successful PBL sessions (Barrows and Tamblyn 1980; Barrows 1985; Maudsley 1999).
PBL requires tutors to work in small groups of 5-10 students. In countries with a low faculty-to-student ratio such as Europe or Asia, tutorials cannot be covered solely by faculty with content expertise in each PBL case scenario as the workload would be too high (Kobayashi 2004; Oda and Koizumi 2008; Yoshioka 2010). To address this weakness, non-content-expert faculty members and sometimes even senior students have been taken on as tutors. A number of studies have investigated the relationship between the tutor’s background and learning outcomes, including the quality of the learning processes (Moust and Schmidt 1994; Maudsley 1999; Solomon and Crowe 2001; Matthes et al. 2002; Kassab et al. 2005a; Moore and Kain 2011), but their conclusions are inconsistent. Some studies show that there are advantages in having content-expert tutors in promoting positive learning outcomes (Davis et al. 1992; Schmidt et al. 1993) while others have found that no such difference exists (Regehr et al. 1995; Dolmans et al. 1996).
A possible reason for this inconsistency may reside in the variability of how the PBL is operated. Each medical school has been developing PBL curriculum based on their own educational context. Appropriate tutoring techniques and attitudes to encourage student group dynamics and self-learning can be different depending on the objectives of the PBL and the students’ maturity as self-directed learners. Therefore, research in a specific educational context is needed for the development of a comprehensive PBL process.
Recently, trials have begun in several medical schools in Japan using senior students as tutors, but few studies have focused on the tutor’s background (Kon et al. 2008; Yoshida and Endo 2008; Suzuki et al. 2012). The objective of our study was to examine the association between tutor performance and related factors, including the tutors’ background, in a Japanese undergraduate medical education setting. We hypothesized that student tutors would be evaluated as highly as content-expert tutors across the PBL curriculum.
In Japan, students who have graduated from high school can apply for entry into the medical school and its 6-year program of study (Onishi and Yoshida 2004; Kozu 2006). The first 4 years are typically allotted to pre-clerkship education, including the liberal arts, basic science, and the theoretical part of clinical medicine. The 4th-year medical students are required to pass a Common Achievement Test as a prerequisite to participate in the clinical clerkship that is offered during the 5th and 6th years (Fadhilah et al. 2011).
In 2002, Saga Medical School introduced a 2-year PBL curriculum as the main teaching strategy for 3rd and 4th year students (Oda and Koizumi 2008). In this restructured curriculum, clinical medicine was integrated with basic medicine, public health and behavioral science, and then divided into 10 organ-system based units (Table 1). A total of 49 PBL sessions were implemented over a 61-week period. Each one-week session was broken into three major steps. In Step 1, tutors facilitated small-group discussions on a paper-based case scenario and promoted the development of a list of learning issues. In Step 2 students focused on independent learning to deepen their understanding of the identified issues. In Step 3, under tutor guidance, students would share their findings and apply this new knowledge to the previously discussed case.
Around 230 faculty members were enrolled as tutors in one of the 10 PBL units and, on average, took charge of a PBL session for 3.4 weeks per year. Faculty tutors without previous PBL experience were trained during a 3-hour shadowing program where they could observe how tutors taught. Before each PBL session, all the faculty tutors attended a briefing session to establish the learning goals and to become familiar with the characteristics of the case scenario.
The student-tutor system was introduced at Saga Medical School in 2008. Students in their 6th and final year who had registered for the elective course “Development of Teaching Competence” joined the PBL program as student tutors. At the beginning of this elective course, they received 90 minutes of tutor training by the chairperson of the PBL curriculum.
Organ-System based units of the PBL curriculum at Saga Medical School.
A cross-sectional survey was performed at Saga Medical School from April 2009 to February 2010 with the approval of the school’s Institutional Review Board. During the research period, PBL sessions with 49 case scenarios were carried out with 3rd- and 4th-year students. At the end of each Step 3 PBL session, students evaluated both their tutor’s performance at Step 1 and 3, and the quality of the case scenario using a 5-point Likert scale (1 = poor, 2 = fair, 3 = good, 4 = very good, and 5 = excellent). Students completed the evaluation forms and posted them to the Office of Student Affairs. They were reassured that their evaluations would not be shared with the tutor or impact on their academic achievement results. The response rate was 95.6%.
ParticipantsA total of 234 tutors and 191 students participated in the PBL sessions. The 94 3rd-year and 97 4th-year students were organized into 15 groups of 6 or 7 for a period of two to three months before being randomly reassigned into another group. Tutors were changed after each full session was completed and every student received equivalent tutoring from either a student or faculty tutor.
The tutors were classified into four groups, according to their backgrounds: student tutors and three faculty categories. The latter consisted of: content-expert clinicians who were involved in clinical work or research at Saga Medical School Hospital which related directly to the PBL case sessions they were to tutor; non-content-expert clinicians whose work was not directly related to the PBL case scenarios; and basic scientists who primarily worked as researchers but were not necessarily qualified medical doctors.
Statistical AnalysisA multivariate logistic regression analysis was used to assess what contribution independent variables had on the dependent variable: the evaluated tutor’s performance. Students’ gender and year level (3rd or 4th), the tutor’s gender and background, and the quality of the case scenario were used as independent variables. The dependent variable, originally assessed on an ordinal 5-point Likert scale, revealed a heavily skewed distribution towards the rating of “Excellent” as shown in Table 2. It was thought that this skewed distribution was due to students adopting a basic point-deduction scoring system where being satisfied with their tutor’s performance resulted in an “Excellent” rating and a lack of satisfaction resulted in low scores. The scores were therefore dichotomized into two categories: “Excellent” or “The others” (fair/ good/ very good). Nine “poor” ratings were statistically rejected after applying the Grubbs-Smirnoff’s test.
The quality of scenario ratings were also dichotomized into two categories: “Excellent” or “other” due to the same issue with skewed distribution. The tutor’s gender and background were modeled as categorical variables. These factors along with the independent variables of being male student, male tutor and a basic scientist tutor were used as reference points. The interaction between student and tutor genders was also entered as a separate independent variable.
Modeling was used to obtain the odds ratios (OR) and their 95% confidence intervals (CIs) to predict how much each independent variable contributed to the dependent variable. Variables with more than 4 variance inflation factors were excluded to avoid multicollinearity. The ‘goodness of fit’ was examined by Akaike’s Information Criterion (AIC). Initially all of the independent variables were entered into the analysis to calculate their AIC and then recalculations were made as each poorly rated variable was removed until the smallest AIC was obtained.
All statistical tests were two-sided, and a p-value of 0.05 was deemed to be of statistical significance. All statistical analyses were conducted using SPSS 20.0 (IBM Corporation, Armonk, New York, USA).
Frequency distribution and percentiles of students’ rating.
To adopt multivariate logistic regression analysis, the ratings of tutor performance was dichotomized into two categories, “Excellent” or “Other” (fair/ good/ very good) after statistically rejecting nine “poor” ratings using the Grubbs-Smirnoff’s test. The quality of scenario ratings were also dichotomized into two categories “Excellent” or “Other”.
Table 3 lists the characteristics of the medical students and the tutors. While there was no major gender imbalance regarding the students, their tutorial groups or their student tutors, 82.5% of the faculty tutors were male. Of these tutors, 82.4% were clinicians and 17.6% were basic scientists. All but 6 faculty tutors had more than a year of PBL tutor experience.
The results of the multivariate logistic regression analysis are summarized in Table 4. Student gender, the quality of case scenarios and particular tutor backgrounds were significantly associated with “Excellent” grades on tutor evaluations. A rating of “Excellent” regarding case scenarios was associated with excellent tutor evaluations (OR of 12.43 [95% CI: 10.28-15.03]). When comparing the tutor’s background, basic scientists, students and content-experts clinician were positively associated with tutor evaluations (ORs of 1.77 [95% CI: 1.15-2.72] and 1.47 [95% CI: 1.11-1.97]), while non-content expert clinician were negatively associated with tutor evaluations (OR of 0.72 [95% CI: 0.55-0.95]). The tutor’s gender did not correlate with the student evaluations, however, the interaction between student and tutor genders showed a positive association (OR of 1.85 [95% CI: 1.10-3.11]): tutor evaluations were higher when both the student and the tutor were of the same gender.
Characteristics of Medical Students and Tutors Participating in PBL Sessions.
Multivariable logistic regression analysis: Odds of obtaining an excellent tutor performance according to junior students’ ratings.
This study found that student gender, the quality of case scenarios and tutor backgrounds were correlated with “Excellent” scores in tutor performance via student evaluations. Though our original hypothesis assumed that the tutors’ background would be a strong predictor for this outcome, the quality of the case scenarios achieved the highest odds ratio (12.43).
The importance of realistic and multidimensional problems in case scenarios has been widely recognized as the key to successful PBL sessions (Dammers et al. 2001; Kenny and Beagan 2004; Steinert 2004; Dolmans et al. 2005; Nieminen et al. 2006). Dolmans et al. (2002), also reported that good problems improve tutor performance. Our research results support these findings.
Concerning tutor backgrounds, student tutors achieved higher ORs (1.77) than faculty tutors regardless of the latter’s specialty. Some researchers have reported similar results and have given useful suggestions to explain why this imbalance may have occurred. Steele et al. (2000), reported that students tended to give student tutors slightly higher evaluations. They also found that no differences existed between faculty tutors and student tutors regarding the knowledge-based examinations or group process work. De Grave et al. (1990), suggested that student tutors might be better able to understand student problems, assess their prior knowledge, and explain concepts using language and examples students understand better than those faculty tutors might use.
Our student tutors might be superior to faculty tutors in creating a good environment for group discussions and in providing more appropriate facilitation because they had, as 4th year students, recently experienced the same PBL sessions as they were now conducting and so may more readily relate to the junior students’ knowledge levels and thought processes. PBL tutoring requires content expertise (knowledge and experience in a clinical setting) and facilitation skills. Appropriate case scenarios including a tutor’s guide are essential to help student tutors overcome any shortages in their content expertise knowledge and to make student tutors’ educational performance more effective.
On a cautionary note, Steele et al. (2000), observed that student-tutor-led groups might take shortcuts in the learning process and Matthes et al. (2002), reported that students’ self-learning time tended to be shorter in such groups. In our research, it is not known whether highly rated tutor performances facilitated active student self-learning or not. Further investigation is needed into the time and quality aspects of the student self-learning process.
The interaction between student and tutor genders was also significant. Kassab et al. (2005b), suggested that understanding gender differences in group behavior is important for PBL programs. Considering that gender difference is ineluctable, further research is needed regarding how it impacts on the PBL process.
Our study has several limitations which are inherent in the evaluation method selected. This pilot study used a global rating about tutor performance and case scenarios. Although this simplicity made it easy to administer the evaluations, it was impossible to evaluate the reliability of the gathered data. Further research focusing on the tutorial process, learning outcomes and an objective evaluation of academic achievement are needed to elucidate students’ reliable evaluation of the PBL sessions. Another limitation of this study is that the generalizability of the findings are limited because the research was conducted in a single educational institution in Japan.
In spite of these limitations, this study revealed what factors are associated with highly rated tutor performance from the students’ point of view. “Excellent” case scenarios were the most important factor followed by the inclusion of 6th year students as tutors in a PBL curriculum. These findings would be useful to other learning institutions that have, or plan to introduce a student tutor system into their teaching curriculums.
We would like acknowledge Ms. Akiko Kimoto and Ms. Miho Ueda for their assistance in gathering data for this study. This work was supported by a Grant-in-Aid for Scientific Research (C) #22590475 from the Japanese Society for the Promotion of Science.
The authors declare no conflict of interest.
