Almost All Machine Learning Algorithms Are A/B Tests

Abstract

Machine learning and other algorithms produce a growing portion of decisions and recommendations. Such algorithmic decisions are unintentional A/B tests since the algorithms make decisions based only on observable input variables. We use this observation to characterize the sources of causal-effect identification for a class of stochastic and deterministic algorithms. Data from almost every algorithm is shown to identify some causal effect. This identification result translates into consistent estimators of causal effects that are easily implemented even with high dimensional data and complex algorithms.