On Regularization Functions for Rule Ensemble Learning

Abstract

We investigate regularization functions for rule ensembles, which have a form of weighted sum of rules. The rule ensembles are regarded as linear classifiers whose variables correspond to rules. Generally, there are tremendous possible rules in a data set. Hence, we should select a small subset of important rules. On the other hand, because of a large number of rules, some of rules correlate with other rules, i.e., the problem of multicollinearity arises. For rule selection, L1 regularization for a vector of rule weights is adequate, however, for multicollinearity, L2 regularization is required. In this study, to overcome these problems, we use a regularization function proposed by the author, which can adjust the size of rules in an ensemble.