Real-Time Eyeglasses Replacement Method for 3D Virtual Try-on

Abstract

This paper presents a 3D virtual eyeglasses try-on system for practical use. For fitting eyeglasses in a shop, consumers wish to look at themselves in a mirror while trying on various eyeglass styles. However, for people who need to wear eyeglasses for correcting problems with eyesight, it is impossible for them to clearly observe their face in the mirror without wearing eyeglasses. This makes fitting them for new eyeglasses difficult. Many studies on virtual glasses try-on have been conducted, and some eyewear brands have already put the system into practical use, but none of these mixed reality systems considered processing in case users wear their own glasses. This research proposes a virtual try-on system that can be used while wearing eyeglasses. We replace the user’s eyeglasses in the input video with new eyeglasses virtually.Also, a fast and accurate face tracking tool enables our system to automatically display 3D virtual glasses following a user’s head motion. Moreover, taking into account the change due to distortion of the correction lens and the refraction and reflection of light, t is possible to generate more realistic glasses by drawing a virtual lens. Experimental results demonstrate that the proposed method can render virtual glasses naturally while the user is wearing real eyeglasses.