Privacy-Preserving Data Annotation Automated by Large Multimodal Model

Abstract

In recent years, large multimodal models(LMMs) have demonstrated innovative performance in various tasks such as text analysis, transcription, and optical character recognition. On the other hand, the data usage policies of LMMs depend on developers and thus there is a potential risk that confidential information could be stored or used as training data. Various LMM applications have been explored in academia and industry, and there is a growing demand for technologies that utilize LMMs while preserving data privacy. One such application of LMMs is the automation of data annotation tasks. Traditional data annotation is performed manually, requiring much time and cost. Also, the quality of the annotation is highly dependent on the individual abilities of each annotator. Therefore, LMMs have been expected to be faster and more accurate annotation resources. In our study, we propose a framework that balances annotation accuracy and data privacy preservation in data annotation tasks. Our experiments employed LMMs for image annotation and showed a reduction in privacy leakage risks while maintaining annotation accuracy.