Rainfall-based warning systems are widely used as a means of evacuating inhabitants to prevent sediment disasters. However, considering that only 2.2% of Japanese local governments carried out evacuation orders after sediment disaster alerts were issued in 2008, it appears that the existing rainfall-based warning systems are neither effective nor taken seriously. Furthermore, the case study of Shiaolin village in Taiwan indicates that the existing rainfall-based warning systems may not be sufficient. In addition, most assessments of the effectiveness of warning systems have merely examined whether alerts were issued before a disaster occurred; the appropriate timing of alerts has not been thoroughly studied. Here, we examine the characteristics of the warning models and warning-issuing systems in Japan and Taiwan. We propose evaluation indices of the effectiveness of warnings, and focus on the shortcomings of rainfall-based warning models through case studies of disasters and several years of statistical data. Finally, we recommend improvements to disaster prevention strategies.