Shot blast devices and shot peening devices are used in surface processing in steel manufacturing. Shot process is used to give micro-distortions to the product surface in order to strengthen surface toughness. However, micro-distortions often produce changes in the magnetic properties of steel objects. Residual magnetization also remains in objects after handling by magnetic lifters. This residual magnetization is not particularly strong but is sufficient to attract shot particles, and some of the attracted particles remain on the object surface. Remaining particles become the obstacles in following processes such as a painting line. To avoid remaining shot particles, the authors developed a shot particle removal device using permanent magnets and carried out magnetic field simulations to optimize the design of this device. Some of the simulations described in this paper show conditions which include the strength and duration of magnetization for particle removal. Other simulations show the optimal position of the permanent magnets and adjustment steel plates to satisfy those conditions. The simulation results show that some shot particles are adsorbed by residual magnetization and remain on the object surface. The authors confirmed effectiveness of the removal device by some experiments.