Exterior finishes such as tiles and mortar construction are applied to the building structures for the purpose of skeleton protection and aesthetics. Concrete pieces and tiles on the outer wall may fall due to initial defects or aging. In order to prevent this, inspections are performed by visual inspection of the appearance or tapping method. These methods have many problems, such as the need for aerial work platforms and the skill level of the inspector affecting the diagnostic accuracy. Therefore, the passive infrared thermography method, which enables remote inspection and recording, is attracting attention. However, in this method, there is a problem that the background reflection component of the sun rays and surrounding buildings causes a decrease in measurement accuracy. In this paper, we proposed a method to remove or suppress background reflection components by restoring the three-dimensional trajectory of an infrared thermography, the distance to the object to be photographed, and the direction of the surface using V-SLAM or SfM. Moreover, in the laboratory experiments, it was confirmed that the background reflection component can be removed or suppressed by applying this method to infrared thermography images with background reflection.