A Sampling Method based on Generalized Relative Square Error to Emphasize Low Probability Events

Abstract

A method of data sampling from a huge data set is discussed. We introduce a generalized relative square error to emphasize low probability events and figure out the best sampling weight to reduce the error. Our arguments are based on the large deviation theory. Large reduction in the generalized relative square error was numerically confirmed for the best sampling weight. We also propose to use Wang-Landau algorithm in data sampling. This algorithm is not only efficient to estimate a distribution of the original data, but also useful in data sampling to suppress the statistical errors.