In Augmented reality (AR) applications, an ideal AR system must maintain consistency between the real world and AR images. In other words, we need to merge virtual images in the real world naturally and present them to the user. In this paper, we give an example of AR application of medical images, introduce the basics of optical seethrough display, and then explain the concept of consistency issues between the real world and virtual contents. The consistency issue is an important requirement for utilizing AR. We introduce each concept while digging in related research in recent years. This paper assumes readers who are interested in applying AR technology to their fields.

The interest in Optical see-through head-mounted displays (OST-HMDs) has been on the rise in the last years. One envisioned application scenario in medical applications is accurate overlay of information onto the patient. To do so, it is necessary to accurately align the virtual content with the real world for each user, also referred to as calibration of an OST-HMD. In this paper I discuss introduce different methods to calibrate an OST-HMD and discuss open questions that must be answered to accurate, interaction-free calibration.

In this research, we develop a synchronized projector-camera system to capture plane-to-ray light transport based on epipolar geometry. Capturing subsurface scattering light passing through human skin and ignoring direct reflection from the surface of the skin, we realize to enhance the visibility of vessel location. This technique is expected to be applied to the medical imaging as a new aspect of augmented reality technology.

The patient-specific CT/MRI data is already used as clinical XR (extended reality) including virtual reality (VR), augmented reality (AR), mixed reality (MR) by medical image analysis for planning and simulation of therapeutic procedures, navigation and training, etc. In recent years, clinical XR application enables to build using inexpensive 3D reconstruction software and web service. The key to developing this is the automatic extraction of organs by artificial intelligence (AI) and deep learning (DL).

Preoperative 3D simulation of hepatectomy has become popular since it was covered by insurance in 2012 and it is now indispensable. A recent 3D analysis workstation is very useful because it is easy to measure liver volume, understanding liver surgical anatomy, portal and hepatic venous irrigation area. Many studies on preoperative 3D simulation of liver surgery have been reported. We have independently developed liver resection simulation software “Liversim” which enables deforming of the liver, and it is useful for real surgery and surgical education. We became able to diagnose and analyze by using 3D images generated from CT and MR, and it became possible to reference simulation images as guides not only during surgical planning before surgery but also intraoperatively. However, it has not yet reached real navigation like car navigation. We are developing a novel surgical navigation system interacting 3D camera and surgical simulation system. This time I will describe the current state of liver simulation surgery and stateof-the-art technology development.

This paper describes the noise removal effect of low count PET images by using various convolutional neural networks (CNN). As the architecture of CNN, we used Remez 's DenoiseNet (DN) and U-Net with residual learning (UR-Net). In the results, streak artifacts were appeared in the coronal plane by using DN. On the other hand, the artifact was reduced by DN-Nch which removes noise by using adjacent N slices as N channel images. Furthermore, UR-Net and UR-Net×2 which is stacked in two layers improved peak signal to noise ratio (PSNR) compared to DN. In addition, it was indicated that a blind noise removal which removes noise with unknown noise level could be possible by training image dataset which is different thinning rates.

The interpretation of liver cancer is visually performed by doctor, but it is time-consuming. Although one of the solutions is to detect the cancer lesion by using a machine learning technique, it requires a massive number of training samples. We thus aim to develop a method to synthesize various artificial cancer images by overlaying real cancer legion on CT images obtained from normal subjects. The method proposed in this paper is for lesion image synthesis considering the size, the shape and the contrast of liver cancer lesion. The synthesized images are used together with real images in order to construct an accurate cancer detector. We evaluated the proposed method through experiments, and confirmed the effectiveness of the proposed method.