In this paper, we tackle anomaly detection in cluttered wide-field images. Typically, the conventional approaches perform patch-wise anomaly detection by cropping the original image. However, it is difficult to detect anomalies correctly when the image has significant variations and local anomalous areas. Therefore, inspired by the human visual inspection, we propose a novel anomaly detection framework called Gaze-based Anomaly Detection (GAD). Our GAD learns a gaze map obtained from inspectors and utilizes the map to pay attention to the anomalous areas. The experiment showed that the proposed method allows us to detect abnormal samples without cropping and localization pre-processing and outperforms the conventional ones.