Robustly estimating heterogeneity in factorial data using Rashomon Partitions

About this event

Many statistical analyses, in both observational data and randomized control trials, ask: how does the outcome of interest vary with combinations of observable covariates? For example, how do various drug combinations affect health outcomes, or how does technology adoption depend on incentives and demographics? Ubiquitous tools (e.g. Bayesian and frequentist regression, regression trees) partition observations into homogeneous ``pools' where the outcome is similar within pool and different across pools, then compute a summary (or fit a model) within each pool. We propose learning these partitions using Rashomon Partition Sets (RPSs). We construct the RPS by enumerating all partitions that have posterior density close to the maximum a posteriori (MAP) partition, even if they offer very different substantive explanations. The RPS incorporates uncertainty amongst partitions with high evidence in the data, in contrast to averaging approaches (e.g. Bayesian Model Averaging, random forests) that include many partitions that a scientist would easily discard after seeing the data. Looking across partitions in the RPS provides robust explanations for heterogeneity that appear in all partitions in the neighborhood of the MAP. Partitions in the RPS follow two guiding principles. First, they must be scientifically coherent, which also improves computational efficiency. Second, they must be robust. We use a $\ell_0$ prior, which we show is minimax optimal, but impose no additional assumptions on the dependence structure. Conditional on being in the RPS, can calculate the posterior of any measurable function of the vector of feature combination effects on outcomes and characterize approximation error relative to the entire posterior. We give three empirical examples: price effects on charitable giving, heterogeneity in chromosomal structure (telomere length), and the introduction of microfinance. We highlight robust conclusions, including affirmations and reversals of extant findings.