In this study, the effect of machined surface layer on residual stress relaxation was investigated. In previous study, low cycle fatigue strength was affected by residual stress. However, residual stress relaxation was complex because local plastic strain occurred by stress near yield stress. Also, machined surface layer affect the yield stress. Therefore, machined surface layer was modeled using the crystal plasticity model, i.e. plasticity model based on crystallographic deformation mechanics. To describe the microstructure of plastic deformation layer, initial dislocation density and back stress near surface were changed. To describe the microstructure of fine grained layer, grain size near surface was changed. Residual stress relaxation was simulated by crystal plasticity finite element method. Three types of machined surface layers were modeled. Two kind of strain amplitude condition was simulated. In simulation results, local plastic strain was occurred under global elastic condition. Residual stress after cyclic load was different from machined surface conditions under low strain loading. Residual stress was largely relaxed in all cases under high strain loading. Comparing fatigue life of experimental results, simulation results were thought to be valid. Therefore, residual stress relaxation is able to be predicted using this model. Hardening in plastic deformation layer prevents yield by tensile load. However, plastic strain was occurred in plastic deformation under compression load. On the other hand, fine grain layer prevent yield by not only tensile but also compression.