Evaluating the Effectiveness of Metacognitive Prompting in Causal Inference Using Large Language Models

Abstract

Causal inference using large language models (LLMs) has become an important research topic in recent years. In addition, research and development on prompt engineering has been actively conducted to improve the accuracy of LLMs responses. In particular, metacognitive prompting that apply human introspective thinking are known to significantly improve response accuracy in various tasks. In this study, we evaluate the effectiveness of metacognitive prompting on necessary/sufficient cause decision problems. The results show that metacognitive prompting was not necessarily effective. On the other hand, it is found that we can lead to the correct answers to the judgment problems which cannot be solved at all by using the metacognitive prompting, by providing multiple examples of similar problems with correct answers.