A non-invasive material testing methodology is investigated for the purpose of material modeling of biological soft tissue in finite element analysis of human body. An identification method of in vivo displacement field is developed as first step of the investigation. Two kinds of voxel models are constructed for initial and loaded body by means of newly developed experimental system with X-ray CT. We pose an identification problem of a mapping function from the original state to deformed one. The initial voxel model is transformed via the mapping function in the form of the Fourier series with tentatively determined coefficients. An error function is evaluated as square-sum of deference between pixel values assigned to voxels at the same position of actually loaded and virtually transformed voxel models. The optimum values of the coefficients are searched for via SCE-UA method (Shuffled Complex Evolution Method Developed at the University of Arizona), since the pixel value is integer and the positions of error evaluation are discrete. An experiment using inhomogeneous specimen demonstrates a validity of the proposed method.