Bayesian Linear Regression with a Sparse Prior: Inference for a Single Coordinate

We investigate inference for a single coordinate in high-dimensional linear regression, focusing on settings where there is a single parameter of interest (such as a treatment effect) in the presence of numerous potential confounding covariates. A common Bayesian procedure in such scenarios is to model the nuisance covariates using a model selection prior, which encourages sparsity in the regression coefficients.

This work characterizes the behavior of the model selection posterior distribution for the coordinate of interest as sample size grows. Under certain conditions, we establish that the posterior marginal distribution achieves optimal inferential properties through a Bernstein-von Mises theorem: it converges to a normal distribution centered at an efficient oracle estimator with optimal variance. Strikingly, such performance is attainable under conditions where other priors or frequentist procedures such as the LASSO require debiasing.

However, we also identify settings where the posterior marginal exhibits problematic limiting behavior; converging to a multimodal mixture with components that contain substantial bias and/or suboptimal variance, resulting in poor coverage. We propose adjustments to the prior specification that provably restore desirable asymptotic properties whenever such issues arise.