Use and Evaluation of Generative Artificial Intelligence by Medical Students in Japan

Abstract

Introduction: Generative artificial intelligence (AI) has become more accessible due to technological advancements. While it can support more efficient learning, improper use may lead to legal issues or hinder self-directed learning. Medical education is no exception, as generative AI has the potential to become a powerful tool. However, its practicality remains uncertain. Therefore, we investigated how generative AI is perceived among medical students and utilized within the realm of medical education.

Methods: In January 2024, we conducted a study with 123 second-year medical students who had completed a physiology course and laboratory training at Gunma University, Japan. Students used ChatGPT (Chat Generative Pre-trained Transformer) 3.5 (OpenAI) for four tasks and evaluated its responses. A survey on the use of generative AI was also conducted. Responses from 117 participants were analyzed, excluding six non-participants.

Results: Among the students, 41.9% had used ChatGPT. The average scores for tasks 1-4 were 6.5, 4.6, 7.4, and 6.2 out of 10, respectively. Although 13% had a negative impression, 54 students found it challenging to apply for medical purposes. However, 64.1% expressed a willingness to continue using generative AI, provided its use extended beyond medical contexts.

Conclusions: Nearly 60% of students had never used generative AI before, which is consistent with general usage trends. Although they were impressed by the speed of generative AI responses, many students found that it lacked precision for medical studies and required additional verification. Limitations of generative AI, such as "hallucinations," were evident in medical education. It remains important to educate students on AI literacy and their understanding of the potential issues that generative AI could bring about.