The aim of this study is to establish a reliable criterion for rating the level of Japanese words, and to develop an automatic word level rating system based on the criterion. We have already developed word level rating systems based on the coverage for the Japanese language proficiency test (Kawamura, 1999), word familiarity, and word frequency (Kawamura and Kitamura, 2008), and released them on our web page (http://language.tiu.ac.jp/) as part of Reading Tutor. However, they are not sufficient enough to rate the level of Japanese words for JSL/JFL learners. We are developing two kinds of word level rating systems based on the inverse document frequency (IDF): calculated from a newspaper database and from Yahoo-Chiebukuro. In the present study, these two systems are evaluated with texts at different levels.