g-prior

In statistics, the g-prior is an objective prior for the regression coefficients of a multiple regression. It was introduced by Arnold Zellner.^[1] It is a key tool in Bayes and empirical Bayes variable selection.^[2][3]

Definition

Consider a data set $(x_1,y_1),\ldots,(x_n,y_n)$, where the $x_i$ are Euclidean vectors and the $y_i$ are scalars. The multiple regression model is formulated as

$y_i = x_i^\top\beta + \varepsilon_i.$

where the $\varepsilon_i$ are random errors. Zellner's g-prior for $\beta$ is a multivariate normal distribution with covariance matrix proportional to the inverse Fisher information matrix for $\beta$.

Assume the $\varepsilon_i$ are iid normal with zero mean and variance $\psi^{-1}$. Let $X$ be the matrix with $i$th row equal to $x_i^\top$. Then the g-prior for $\beta$ is the multivariate normal distribution with prior mean a hyperparameter $\beta_0$ and covariance matrix proportional to $\psi(X^\top X)^{-1}$, i.e.,

$\beta |\psi \sim \text{MVN}[\beta_0,g\psi(X^\top X)^{-1}].$

where g is a positive scalar parameter.

Posterior mean of $\beta$

The vector of regression coefficients $\beta$ can be estimated by its posterior mean under the g-prior, giving the closed form expression

$\hat\beta = \Big(g\psi(X^\top X)^{-1}+\psi I\Big)^{-1}y.$

Selection of g

Estimation of g is slightly less straightforward than estimation of $\beta$. A variety of methods have been proposed, including Bayes and empirical Bayes estimators.^[3] As g →∞, we would obtain posterior estimate the same as frequentist estimate.

References

1. Zellner, A. (1986). "On Assessing Prior Distributions and Bayesian Regression Analysis with g Prior Distributions". In Goel, P.; Zellner, A. Bayesian Inference and Decision Techniques: Essays in Honor of Bruno de Finetti. Studies in Bayesian Econometrics 6. New York: Elsevier. pp. 233–243. ISBN 0-444-87712-6.
2. George, E.; Foster, D. P. (2000). "Calibration and empirical Bayes variable selection". Biometrika 87 (4): 731–747. doi:10.1093/biomet/87.4.731.
3. Liang, F.; Paulo, R.; Molina, G.; Clyde, M. A.; Berger, J. O. (2008). "Mixtures of g priors for Bayesian variable selection". Journal of the American Statistical Association 103 (481): 410–423. doi:10.1198/016214507000001337.