This paper addresses the use of nonlinear system identification in the mechanical modeling of a high-damping rubber damper. The model consists of one linear spring, two linear dashpots, and one nonlinear spring in order to represent the amplitude- and frequency-dependent properties of the damper. The mechanical properties of the linear spring and two dashpots are determined from the test results by using the nonlinear system identification technique. The restoring force characteristic of the nonlinear spring is formulated by incorporating a versatile hysteresis model. The validity of the model is shown by comparing the test results with the analytical results via the model. Seismic response analyses are carried out for a building with the high-damping rubber dampers using the model, and this demonstrates their effectiveness.