Longitudinal PBL in Undergraduate Medical Education Develops Lifelong-Learning Habits and Clinical Competencies in Social Aspects

Abstract

Problem-based learning (PBL) is popular in medical education in Japan. We wished to understand the influence of PBL on the clinical competence of medical residents, using self-assessment and observer assessment. Tokyo Women’s Medical University (TWMU) implemented PBL longitudinally (long-time) for four years, and on this basis we analyzed whether long-time PBL education is useful for clinical work. A self-assessment questionnaire was sent to junior and senior residents who were alumni of several schools, and an observation-based assessment questionnaire to senior doctors instructing them. Respondents were asked if they had used the PBL process in daily clinical tasks, and if so in what processes. Senior doctors were asked whether TWMU graduates perform differently from graduates of other schools. TWMU graduates answered “used a lot” and “used a little” with regard to PBL at significantly higher rates than other graduates. As useful points of PBL, they mentioned extracting clinical problems, solving clinical problems, self-directed leaning, positive attitude, collaboration with others, presentation, doctor-patient relations, self-assessment, and share the knowledge with doctors at lower levels and students. Observer assessments of TWMU graduates by senior doctors represented them as adaptive, good at presenting, good at listening to others’ opinions, practical, selfish, and eager in their instructional practice. Longitudinal PBL can be a good educational method to develop lifelong-learning habits and clinical competencies especially in terms of the social aspect.

Introduction

Medical education in Japan changed rapidly in the last decade of the 20^th century in response to changes in health conditions and social needs (Onishi and Yoshida 2004; Kozu 2006; Suzuki et al. 2008). Problem-based learning (PBL; also called tutorial education) (Barrows and Tamblyn 1980; Wood 2003) had been part of medical education in Japan since 1990 (Kozu 2006). Integrated courses using PBL for education in basic medicine and clinical medicine have been introduced in many schools, and PBL has been systematically incorporated into an integrated organ- and system-based curriculum in many medical schools. In this curriculum, students learn how to identify and solve problems in a student-centered active learning style delivered by PBL in small groups. At a recent date, 70 (AJMC, Association of Japanese Medical Colleges 2009) to 74 (AJMC 2011) of 80 medical schools in Japan, 51 public and 29 private, were implementing PBL in undergraduate education (JSME, Japan Society for Medical Education 2014) alongside lectures and practical training activities. All medical schools in Japan accept students who have completed a high school diploma or higher qualification. The programs are six years long; upon graduation, students receive a Bachelor of Medicine degree, which enables them to sit the national medical board examination. Those who pass the exam complete an initial two-year postgraduate residency program, called “core medical training,” mainly focused in primary care and general medicine, followed by a four-year residency program, called “specialty training.”

Tokyo Women’s Medical University (TWMU) introduced PBL in 1990, as the first medical school in Japan to do so. Before the introduction of PBL in 1990 lectures and practical trainings were the only educational methods at TWMU, similar to other Japanese medical schools. The university is private and was founded in 1900 in the spirit of improving women’s social status through education that would help them acquire social and financial independence and contribute to society. All the medical students at TWMU are female. The concept of PBL is viewed at TWMU as involving learning how to think like a doctor through case-based learning, self-directed learning, and lifelong learning. Integrated PBL is designed and applied to educate students in problem-finding, logical thinking, and problem-solving. Students start with an introductory PBL on “learning what to know”; then, gradually, the learning objectives change to more clinical and practical problem-solving. Between the first and the fourth year of the six-year medical program, students learn basic science, basic medicine, and clinical medicine successively with topics such as Integrated Basic Human Body, Integrated Human Organs, and Functional Systems; learning takes place through a mix of lectures, PBL, and practical laboratory training (Albanese and Mitchell 1993; Yoshioka et al. 2005; Albanese 2010). In the first four years, students attend PBL sessions twice a week throughout the year. In each session, students spend more than 100 minutes with tutors. At the end of the fourth year students attend Team-based learning (TBL) to learn clinical reasoning in the class (Okubo et al. 2012). In the last two years, students enter the clinical clerkship program, where they spend all day in the hospital.

Based on The Present State of the Medical Education Curriculum 2011, published by the national Association of Japan Medical Colleges Council (AJMC 2012) and a confirmatory supplement from one medical school whose PBL hours were miscounted, the number of medical schools implementing PBL for each year of medical school is shown in Fig. 1A. In the third and fourth years, more than half of medical schools used PBL in undergraduate education. Total PBL tutorial hours over six years are different between schools (Fig. 1B), from 15.0 h to 540.5 h, with an average of 93.9 h. PBL tutorial hours at TWMU were the third-longest. The numbers of medical schools implementing PBL tutorial in four different years, three years, two years, and one year were respectively 7, 17, 28, and 18. The three schools with the longest PBL hours for each year are shown in Table 1. As indicated above, many schools implement PBL tutorials during the third and fourth years. There is no data on the details of learning objectives of PBL tutorial.

However, PBL has come under criticism for being costly and of uncertain educational effectiveness (Smits et al. 2002). Some authors conclude that PBL might in fact be detrimental to cognitive knowledge and reasoning patterns (Albanese and Mitchell 1993; Berkson et al. 1993; Smits et al. 2002).

To explore the influence of PBL in undergraduate medical education upon medical residents’ (later) clinical competence and to explore whether residents’ clinical competencies are affected by a longitudinal PBL like that at TWMU, we analyzed self-assessments by junior and senior residents of their PBL experience and clinical competence. Moreover, we analyzed the results of observation-based assessment of residents by senior doctors in order to compare clinical competencies between residents who had graduated from TWMU (longitudinal-PBL graduates) and those from the other medical schools.

Fig. 1.

The state of PBL in 2011.

(A) Number of schools implementing PBL each year in Japan. (B) Total PBL hours over six years in Japanese medical schools.

Table 1.

The three medical schools with the longest PBL hours for each year.

Methods

Questionnaire

Between February and March 2014 a self-assessment questionnaire was sent to junior residents (first year and second year of residency, called “core medical training,” or CT 1 and CT 2) and senior residents (the third to sixth years of residency, called “specialty training”; that is, ST 3-6), and to senior doctors who were instructing the residents, as an observed assessment. The questionnaire survey was anonymous.

Questionnaire for junior and senior residents: TWMU has ten affiliated hospitals or institutes. A questionnaire was sent to all the junior and senior residents working for these hospitals based on the database of the Personnel Department, via intra-facility mail. For residents who had graduated from TWMU and were working for other hospitals based on the database of the alumni association of TWMU, a questionnaire was sent via post to the home address. PBL alumni were asked if they had used the PBL process in daily clinical tasks, what kind of processes had been influenced by PBL, when they felt their PBL experiences had been useful, whether they had liked PBL, what aspects of PBL were positive or negative compared with lectures, whether they felt PBL was a good educational tool, and whether they would recommend that applicants enter medical schools delivering problem-based education (Table 2).

Questionnaire for senior doctors: A questionnaire was sent to all the senior doctors working for the affiliated hospitals, based on the database of the Personnel Department, who were responsible for instructing residents, via intra-facility mail. They were asked whether junior and senior residents who graduated from TWMU were different than they themselves had been in the old days (Table 3). They were also asked if residents from TWMU were different from residents who had graduated from other medical schools. Respondents who indicated that TWMU residents were different on either of these questions were then asked about the specific differences.

Data analysis: Answered questionnaires were scanned and the data gathered using CARS V8.0 (i4 Corp.). In order to examine whether rates of selection for given list items were different among groups, answers obtained from TWMU graduates were compared with those from other schools’ graduates. Each answer was analyzed with Pearson’s chi-squared test, using JMP 10 (SAS Institute Inc.). Answers from senior doctors were also analyzed by the same method.

Research ethics: This study’s protocol was approved by the Institutional Review Board of Tokyo Women’s Medical University (#3287).

Table 2.

Questionnaire for junior and senior residents.

Table 3.

Questionnaire for senior doctors.

Results

Questionnaire

Resident questionnaire: This questionnaire was distributed to all of the junior or senior residents who had graduated from TWMU between 2008 and 2013: 612 residents (298 (CT 1-2: 95, ST 3-6: 203) working for TWMU hospitals and 314 (CT 1-2: 107, ST 3-6: 207) for other hospitals (Table 4). Among the 612 residents, 191 graduates answered the questionnaire (133 from TWMU hospitals and 58 from other hospitals); thus, the answer rate was 31.2% (44.6% from TWMU hospitals and 18.5% from others). The same questionnaire was distributed to 458 residents (male CT 1-2: 66, female CT 1-2: 18, male ST 3-6: 289, female ST 3-6: 85) who worked for TWMU hospitals, of whom 142 answered (male CT 1-2: 26, female CT 1-2: 7, sex-unknown CT 1-2: 3, male ST 3-6: 45, female ST 3-6: 18, sex-unknown ST 3-6: 1; male, status unknown: 21, female, status unknown: 9, both sex and status unknown: 12) (total answer rate 31.3%). Whereas all of the TWMU graduates had experienced 301 hours of PBL in undergraduate education, only 121 residents from other medical schools had done so (85.2%) with average PBL hours of 95.2 ± 61.3 over six years.

All the residents who had experienced PBL in undergraduate education were asked if they had used the PBL process in daily clinical tasks. Among TWMU graduates, the proportions answering “used a lot” and “used a little” were significantly higher than among other graduates (20.4% vs. 12.4%, p < 0.05, and 64.4% vs. 50.4%, p < 0.0001, respectively). In contrast, in the other-graduate group, the proportions answering “used not much” and “used not at all” were higher than those in the TWMU group (27.3% vs. 13.1%, p < 0.05, and 9.9% vs. 1.6%, p < 0.005, respectively). Among the female residents from other schools (n = 31) the proportion answering “used not much” (32.2%) was higher than that in the TWMU group (p < 0.01).

Residents who answered “used PBL a lot” or “used PBL a little” were asked what kinds of processes were influenced by PBL, in a multiple-choice item listed in Fig. 2. As shown in Fig. 2, in both groups, the top answers were “extracting clinical problems” (53.4% in TWMU vs. 40.5% in others, p < 0.005), “solving clinical problems” (39.8% vs. 24.0%, p < 0.005, 12.9% in female others, p < 0.005 vs. TWMU), and “self-directed leaning” (36.6% vs. 19.0%, p < 0.005); as can be seen, these three tasks were more frequently chosen by TWMU graduates than others. Also, TWMU group, as compared to the other group, more often selected “positive attitude” (17.8% vs. 9.1%, p < 0.01), “collaboration with others” (18.3% vs. 2.5%, p < 0.005), “doctor-patient relation” (11.0% vs. 0.8%, p < 0.005, 0% in female others, p < 0.05), and “sharing knowledge with doctors at lower levels and students” (5.2% vs. 0%, p < 0.005). Finally, “presentation” [12.6% vs. 13.2%, not significant (N.S.)] and “self-assessing” (9.4% vs. 5.0%, N.S.) were moderately selected in both groups.

When asked when they had felt PBL experience was useful, in another multiple-choice item, most respondents mentioned “(during) the first year of residency” (core medical training, or CT 1; 46.1% in TWMU vs. 28.1% in others, p < 0.005, 25.8% in female others, p < 0.05 vs. TWMU), followed by “the clinical clerkship” (38.2% vs. 24.8%, p < 0.005) and “the second year of residency” (CT 2) (25.7% vs. 13.2%, p < 0.005, 6.5% in female others, p < 0.05). The choice of “specialty training” (ST 3-6) was also higher in the TWMU group than in another group (18.8% vs. 13.2%, p < 0.05).

When asked whether they had liked PBL in their school days, both groups most frequently answered “liked a little” (66.0% in TWMU vs. 41.3% in others, p < 0.0001, 38.7% in female others, p < 0.005 vs. TWMU). The answer “liked very much” was obtained more from TWMU group compared with the female other group (16.0% vs. 3.2%, p < 0.05). The answer “disliked a little” was obtained more in the other group (30.6%, p < 0.01) and in the female other (38.7%, p < 0.005) than in the TWMU group (15.7%).

Residents who had experienced PBL were asked to identify positive and negative aspects of PBL as compared with lectures. As positive points, both groups mentioned “(developing) problem-extracting ability” (48.7% in TWMU and 45.5% in others, N.S.), “problem-solving ability” (38.2% vs. 30.6%, N.S.), and “group study (study actively in a group)” (25.7% vs. 29.8%, N.S.) (Fig. 3). In the TWMU group, “(learning) collaboration” (40.3%), “memorable learning (of objectives)” (34.6%), “passive listening” (34.0%), “self-learning” (33.5%), “explanation skill” (20.9%), “motivation to learn” (14.7%), and “activities rewarding free-thinking” (14.7%) were more frequently selected compared with the other group (18.2%, p < 0.05; 20.7%, p < 0.005; 20.7%, p < 0.005; 21.5%, p < 0.01 (12.9% in female others, p < 0.05); 7.4%, p < 0.005; 8.3%, p < 0.01 (3.2% in female others, p < 0.05); and 5.8%, p < 0.01, respectively). As negative aspects of PBL, both groups mostly selected “group-member-dependence of learning” (54.5% in TWMU group and 48.8% in the other group, p < 0.05) followed by “tutor-dependence of learning” (44.0% vs. 43.8%., N.S.) (Fig. 4). Finally, a much greater number of TWMU graduates answered “accept lazy study habits” meaning they were less motivated to study compared with the other group (51.3% vs. 28.9%, p < 0.005).

When asked to evaluate the PBL education, “good” was the most frequent answer in both groups (63.9% in TWMU vs. 58.7% in others, p < 0.01) followed by “very good” (22.5% vs. 15.7%, p < 0.05, 6.5% in female others, p < 0.05 vs. TWMU). When asked whether they would recommend that applicants enter problem-based medical schools or recommend that they not do so, the majority answered “neither” (51.3% in TWMU vs. 54.5% in other, N.S.). However, in the TWMU group more respondents than in the other group answered “recommend” (36.6% vs. 26.4%, p < 0.005), while “not recommend” was less frequent (6.8% vs. 19.0%, p < 0.05).

Senior doctors’ questionnaire: A questionnaire was distributed to 1,675 senior doctors who were involved in instructing residents in TWMU hospitals (485 graduated from TWMU; 1,012 male, other schools; 178 female, other schools; 570 doctors responded (answer rate 33.0%) (Table 4). One-third of the respondents had graduated from TWMU (33.7%), and the others, from other medical schools; senior doctors in TWMU hospitals represented institutions in similar proportions.

When asked whether average junior and senior residents who had graduated from TWMU were different from themselves in the old days, the majority of senior doctors answered in the affirmative (“very different,” 31.9%; “different,” 34.9%; “similar,” 20.5%; “the same,” 1.3%). Among the senior doctors who had graduated from other schools, the proportion of “very different” was higher than that among TWMU-graduated senior doctors (36.9% vs. 27.1%, p < 0.05). Respondents who answered “very different” or “different” were asked to express the nature of the differences, again on a multiple-choice basis. They said that TWMU students were “inactive” (p < 0.005), “good at presentation” (p < 0.005), “adaptive” (p < 0.005), “practical” (p < 0.005), “selfish” (p < 0.005), that they “listen[ed] to others’ opinions” (p < 0.05), and that they were “self-confident” (p < 0.01) (Fig. 5).

Asked whether there were any differences between residents from TWMU and other schools, senior doctors most frequently answered “similar” (38.4%), followed by “different” (32.8%), “very different” (7.0%), and “the same” (1.6%). They then indicated in more detail that the TWMU residents were more “adaptive” (p < 0.005), “good at presentation” (p < 0.005), that they “listen[ed] to others’ opinions” (p < 0.005), that they were “practical” (p < 0.005), “selfish” (p < 0.005), “eager to share their knowledge with doctors at a lower level or students” (p < 0.01), and that they “talk a lot” (p < 0.05) (Fig. 6). Both “active” and “inactive” were selected equally. Among the senior doctors who had graduated from TWMU, the proportions of “selfish” and “self-confident” were higher than those among senior doctors who had graduated from other schools (12.0% vs. 6.8%, p < 0.05, 10.9% vs. 5.4%, p < 0.05, respectively).

Table 4.

Details of questionnaire.

Fig. 2.

Clinical tasks that were influenced by PBL experiences.

Fig. 3.

Evaluation of PBL education by residents: positive points of PBL compared with lectures.

Fig. 4.

Evaluation of PBL education by residents: negative points of PBL compared with lectures.

Fig. 5.

Impressions of TWMU graduates among senior doctors compared with “the old days”.

Fig. 6.

Impressions of TWMU graduates by senior doctors compared with other graduates.

Discussion

Since PBL was introduced in Japanese medical education, several educational effects have been ascribed to PBL: improved motivation to study among students, greater motivation to teach among teachers, elevation of the percentage of students attending lectures, and more self-study (PBL Tutorial Committee, TWMU School of Medicine 2009; JSME 2014). These reports are broadly congruent with reports from other countries showing high satisfaction among students and teachers, and higher student self-assessment results for self-learning, problem-solving, information-gathering, and job performance (Smits et al. 2002). In Harvard Medical School students were randomly assigned to the New Pathway curriculum consisted of PBL, lectures, and labs and to the traditional curriculum consisted of lectures and labs. Students in the PBL curriculum preferred active learning, demonstrated greater psychosocial knowledge, better relational skills and more humanistic attitudes (Moore et al. 1994). There is also a report which shows a positive effect of PBL education on United States Medical Licensing Examination (USMLE) scores (Hoffman et al. 2006), but many reviews conclude that there is no evidence of gained knowledge (Thomas 1997; Smits et al. 2002; Bate et al. 2014).

Apart from the effects on students and teachers, whether this educational approach influences clinical skills in the residential training period is another interest. In the present study, more than half of residents expressed that the experience of PBL in undergraduate education had been helpful for their daily clinical work, especially in their early stages as clinicians. Not only problem-extracting and -solving, which are focused on in the PBL process, but also self-directed learning, collaboration with others, positive attitude, presentation, and fostering doctor-patient relationships were selected by residents as positive impacts of PBL on clinical work. These findings stand alongside previous results showing better clinical performance among PBL graduates than other graduates in areas including developing clinical reasoning, self-learning skills (Schmidt et al. 2006), information gathering (Koh et al. 2008), and better readiness to work as clinicians (Bate et al. 2014). Harvard Medical School reported that compared with traditional curriculum graduates more PBL graduates selected primary care or psychiatry and they felt more confident in the ability to manage patients with psychosocial problems (Peters et al. 2000). Similar to our results PBL graduates liked the educational approaches of their curriculum more than traditional graduates did. Koh et al. (2008) reviewed the effects of PBL on physician competency and conclude that competencies showing strong evidence of improvement based on self-assessment and observed assessment are those in social and cognitive dimensions, such as application of legal and ethical aspects of health care, communication skills, coping with uncertainty, and self-directed or continuing learning. In the present study, many residents who had graduated from PBL-implementing medical schools evaluated PBL as a better educational method compared with lectures, not only in terms of aspects overtly related to the PBL process but also in social aspects, such as acquiring self-learning ability, learning how to collaborate, passive listening, and explanation skills.

Another effect of longitudinal PBL tutorial was investigated by comparing the resident responses between TWMU graduates and graduates of other schools. Our data show more positive responses from TWMU graduates, who are all female, as opposed to the other graduates, who are half male. Generally residents from TWMU tend to evaluate PBL better and to say that they liked PBL in their schooldays. In this light, a limitation of this study is the possibility of bias caused by tolerance of PBL system in TWMU graduates, presumably related to the features of PBL they experienced during their education (see next paragraph). However, residents from TWMU also mentioned specific advantages related to social aspects of group work related to the collaboration process through PBL, including passive listening and explaining to others, more frequently. Not only self-assessment, but also observer assessment by senior doctors indicates that TWMU graduates are good at listening to others and at presentation.

The results of the questionnaire for senior doctors may be affected by various factors. Their comparison of current residents graduating from TWMU with themselves in the old days may be influenced by changes in time and social conditions, residential program, business environment, generational practical sense, and so on. Even though, senior doctors evaluated that TWMU graduates are adaptive, and good at listening others and at presentation compared either with themselves in the old days and with residents from other schools.

Currently, a great number of medical schools implement PBL; however, only around one-tenth of them, including TWMU, perform PBL over four years of medical education. Similar to the situation in other countries, PBL as used in Japan has been modified by each individual medical school. More than 65% of schools implement PBL for one or two years, most often the third and fourth years of schooling. The TWMU curriculum consisting of four years of longitudinal PBL is rare, especially the use of long PBL hours in the first grade to help students learn how to learn. Many TWMU graduates assessed PBL as a memorable learning tool, indicating that this approach seems to be effective by repeating sessions and gradually changing contents more complicated.

In conclusion, the present findings show the effects of longitudinal PBL in Japan and seem to support the idea that longitudinal PBL can be a good educational method to develop not only strong lifelong learning habits, but also clinical competencies in social aspects of medical work, such as self-directed learning ability, learning how to collaborate, passive listening, and explanation skills, especially for clinicians at the early stage.

Acknowledgments

We acknowledge Drs. Noriko Ujihara, Akiyo Tanabe, and Yukie Yamamura for their assistance in designing the questionnaire. We would also like to acknowledge Messrs. Atsushi Ishida, Tetsuo Ishida, Kazuo Ooe, and Toshinori Sasaki and Mses. Tomoe Miyazawa and Megumi Tamagawa for their assistance in distributing and collecting the questionnaire. Finally, we acknowledge Mses. Yumiko Nurishige and Miki Togawa, for their assistance in preparing materials and conducting data analysis.

This study was supported by a Grant-in-Aid for Scientific Research from the Japan Society for the Promotion of Science (JSPS), Tokyo, KAKENHI Grant Number 25460635 and a grant from the Support Program for Distinctive University Education, for which it was selected by the Japan University Accreditation Association (JUAA).

Conflict of Interest

The authors declare no conflict of interest.

References

•  Albanese,  M.A. (2010) Problem-based learning. In Understanding Medical Education: Evidence, Theory and Practice, 1st ed., edited by Swanwick, T. Wiley-Blackwell, Oxford, pp. 37-52.
•  Albanese,  M.A. &  Mitchell,  S. (1993) Problem-based learning: a review of literature on its outcomes and implementation issues. Acad. Med., 68, 52-81.
• Association of Japanese Medical Colleges (AJMC) (2009) White Paper on Medical Education 2009, Association of Japanese Medical Colleges, Tokyo.
• Association of Japanese Medical Colleges (AJMC) (2011) White Paper on Medical Education 2010, Association of Japanese Medical Colleges, Tokyo.
• Association of Japanese Medical Colleges (AJMC) (2012) The Present State of the Medical Education Curriculum 2011, Association of Japanese Medical Colleges, Tokyo.
•  Barrows,  H.S. &  Tamblyn,  R.M. (1980) Problem-Based Learning: An Approach to Medical Education, 1st ed., Springer Publishing Company, New York, NY.
•  Bate,  E.,  Hommes,  J.,  Duvivier,  R. &  Taylor,  D.C. (2014) Problem-based learning (PBL): getting the most out of your students. Their roles and responsibilities: AMEE Guide No. 84. Med. Teach., 36, 1-12.
•  Berkson,  L. (1993) Problem-based learning: have the expectations been met? Acad. Med., 68, S79-88.
•  Hoffman,  K.,  Hosokawa,  M.,  Blake,  R. Jr.,  Headrick,  L. &  Johnson,  G. (2006) Problem-based learning outcomes: ten years of experience at the University of Missouri-Columbia School of Medicine. Acad. Med., 81, 617-625.
• Japan Society for Medical Education (JSME) (2014) White Paper on Medical Education 2014, Shinoharashinsha Co., Tokyo, Japan.
•  Koh,  G.C.,  Khoo,  H.E.,  Wong,  M.L. &  Koh,  D. (2008) The effects of problem-based learning during medical school on physician competency: a systematic review. CMAJ, 178, 34-41.
•  Kozu,  T. (2006) Medical education in Japan. Acad. Med., 81, 1069-1075.
•  Moore,  G.T.,  Block,  S.D.,  Style,  C.B. &  Mitchell,  R. (1994) The influence of the New Pathway curriculum on Harvard medical students. Acad. Med., 69, 983-989.
•  Okubo,  Y.,  Ishiguro,  N.,  Suganuma,  T.,  Nishikawa,  T.,  Takubo,  T.,  Kojimahara,  N.,  Yago,  R.,  Nunoda,  S.,  Sugihara,  S. &  Yoshioka,  T. (2012) Team-based learning, a learning strategy for clinical reasoning, in students with problem-based learning tutorial experiences. Tohoku J. Exp. Med., 227, 23-29.
•  Onishi,  H. &  Yoshida,  I. (2004) Rapid change in Japanese medical education. Med. Teach., 26, 403-408.
• PBL Tutorial Committee, TWMU School of Medicine (2009) Tutorial Education—a New Creation and Implementation—a New Edition, Shinoharashinsha Co., Tokyo, Japan.
•  Peters,  A.S.,  Greenberger-Rosovsky,  R.,  Crowder,  C.,  Block,  S.D. &  Moore,  G.T. (2000) Long-term outcomes of the New Pathway Program at Harvard Medical School: a randomized controlled trial. Acad. Med., 75, 470-479.
•  Thomas,  R.E. (1997) Problem-based learning: measurable outcomes. Med. Educ., 31, 320-329.
•  Schmidt,  H.G.,  Vermeulen,  L. &  van der Molen,  H.T. (2006) Longterm effects of problem-based learning: a comparison of competencies acquired by graduates of a problem-based and a conventional medical school. Med. Educ., 40, 562-567.
•  Smits,  P.B.,  Verbeek,  J.H. &  de Buisonje,  C.D. (2002) Problem based learning in continuing medical education: a review of controlled evaluation studies. BMJ, 324, 153-156.
•  Suzuki,  Y.,  Gibbs,  T.,  & Fujisaki,  K. (2008) Medical education in Japan: a challenge to the healthcare system. Med. Teach., 30, 846-850.
•  Wood,  D.F. (2003) Problem based learning. BMJ, 326, 328-330.
•  Yoshioka,  T.,  Suganuma,  T.,  Tang,  A.C.,  Matsushita,  S.,  Manno,  S. &  Kozu,  T. (2005) Facilitation of problem finding among first year medical school students undergoing problem-based learning. Teach. Learn. Med., 17, 136-141.