We propose a double-referential holographic memory (DRHM) that can detect phase modulation signal by just entering reference beam into recording medium without additional system for generation and irradiation of the beam to interfere with signal beam. In this technique, by highly-accurate and simplified complex amplitude detection, high multi-level phase modulation signal and spatial quadrature-amplitude modulation (SQAM) signal become available, so it is expected that storage capacity and data transfer rate are greatly increased. In this paper, we describe the basic operation of the DRHM and evaluate multiple recording/reconstruction characteristics by numerical simulation. Besides, the method for furthermore improving reconstruction quality is newly proposed and its effectiveness is demonstrated.