This study evaluates action difficulty of residents for three types of high-rise residential buildings, i.e., earthquake-resistant, seismic response controlled, and seismically isolated buildings. The results of questionnaire surveys on action difficulty during earthquakes are reviewed with reference to aseismic types of buildings. A new formula is proposed for action difficulty evaluation using the accumulated values of time history waveforms based on strong-motion records and the questionnaire surveys. It is applied to action difficulty evaluation for three types of building models using seismic records and is validated in comparison with the questionnaire surveys.