How to Evaluate Biases in Financial Investment Decision-Making within Large Language Models ?

Abstract

In recent years, the establishment of new authorized corporations and the introduction of the new NISA system have led to increased interest in financial education across various age groups. Concurrently, there is an expected rise in the use of large language models (LLMs) in services that support financial education, such as chatbots and robo-advisors. However, LLMs are often regarded as 'black boxes', raising concerns about biases in their outputs, including potential racial discrimination. Therefore, in this study we develop metrics to measure and evaluate the biases of LLMs within the context of financial education. Drawing from behavioral economics, we have developed methods to assess LLMs in terms of risk preference, time preference, and social preference. Furthermore, we evaluate various LLMs, including ChatGPT and PaLM, using these proposed metrics and present our findings.