There is a large literature explaining why AdaBoost is a successfulclassifier. The literature on AdaBoost focuses on classifier margins andboosting's interpretation as the optimization of an exponential likelihoodfunction. These existing explanations, however, have been pointed out to beincomplete. A random forest is another popular ensemble method for which thereis substantially less explanation in the literature. We introduce a novelperspective on AdaBoost and random forests that proposes that the twoalgorithms work for similar reasons. While both classifiers achieve similarpredictive accuracy, random forests cannot be conceived as a directoptimization procedure. Rather, random forests is a self-averaging,interpolating algorithm which creates what we denote as a "spikey-smooth"classifier, and we view AdaBoost in the same light. We conjecture that bothAdaBoost and random forests succeed because of this mechanism. We provide anumber of examples and some theoretical justification to support thisexplanation. In the process, we question the conventional wisdom that suggeststhat boosting algorithms for classification require regularization or earlystopping and should be limited to low complexity classes of learners, such asdecision stumps. We conclude that boosting should be used like random forests:with large decision trees and without direct regularization or early stopping.
translated by 谷歌翻译