Spike-and-slab variable selection

Spike-and-slab regression is Bayesian variable selection technique which is particularly useful when the amount of possible predictors is bigger than the amount of observations.[1]

Initially, the idea of the spike-and-slab model was proposed by Mitchell & Beauchamp (1988).[2] The approach was further significantly developed by Madigan & Raftery (1994)[3] and George & McCulloch (1997).[4] The final adjustments to the model were done by Ishwaran & Rao (2005).[5]

Model description

Suppose we have P possible predictors in some model. Vector γ has a length equal to P and consists of zeros and ones. This vector indicates whether a particular variable is included in the regression or not. If no specific prior information on initial inclusion probabilities of particular variables is available, a Bernoulli prior distribution is a common default choice.[6] Conditional on a predictor being in the regression, we identify a prior distribution for the model coefficient, which corresponds to that variable (β). A common choice on that step is to use a Normal prior with mean equal to zero and a large variance calculated based on (XT X)−1 matrix (where X is a design matrix of explanatory variables of the model).[7]

A draw of γ from its prior distribution is a list of the variables included in the regression. Conditional on this set of selected variables, we take a draw from the prior distribution of the regression coefficients (if γi = 1 then β<suub>6 = 0 and if γi = 0 then β = 0). βγ denotes the subset of β for which γi = 1. In the next step, we calculate a posterior probability distribution for both inclusion and coefficients by applying a standard statistical procedure.[8] All steps of the described algorithm are repeated thousands of times using Markov chain Monte Carlo (MCMC) technique. As a result, we obtain a posterior distribution of γ (variable inclusion in the model), β (regression coefficient values) and the corresponding prediction of y.

The model got its name (spike-and-slab) due to the shape of the two prior distributions. The "spike" is the probability of a particular coefficient in the model to be not zero. The "slab" is the prior distribution for the regression coefficient values.

An advantage of Bayesian variable selection techniques is that they are able to make use of prior knowledge about the model. In the absence of such knowledge, some reasonable default values can be used: "For the analyst who prefers simplicity at the cost of some reasonable assumptions, useful prior information can be reduced to an expected model size, an expected R2, and a sample size ν determining the weight given to the guess at R2."[6] Some researchers suggest the following default values: R2 = 0.5, ν = 0.01, and π = 0.5 (parameter of a prior Bernoulli distribution).[6]

A possible drawback of the Spike-and-Slab model can be its mathematical complexity (in comparison to linear regression). A deep understanding of this model requires sound knowledge in stochastic processes. On the other hand, some modern statistical software (e.g. R-program) have ready-to-use solutions for calculating various Bayesian variable selection models.[9][10][11] In this case, it would be enough for a researcher to know the idea of the method, required model parameters and input variables. The analysis of the model outcomes (distribution of γ, β, and corresponding predictions of y) can be more challenging in comparison to linear regression case. The spike-and-slab model produces inclusion probabilities for each of possible predictors. This can cause difficulties when comparing results to the studies with simple regression (usually only regression coefficients with corresponding statistics are available).

Spike-and-slab regression is a part of Bayesian structural time series model (which is used for feature selection, time series forecasting, nowcasting, inferring causation, and other).

See also

References

  1. ↑ "Big Data: New Tricks for Econometrics" (PDF).
  2. ↑ "Bayesian variable selection in linear regression" (PDF).
  3. ↑ "Model selection and accounting for model uncertainty in graphical models using occam’s window" (PDF).
  4. ↑ "Approaches for Bayesian variable selection" (PDF).
  5. ↑ "Spike and slab variable selection: frequentist and Bayesian strategies" (PDF).
  6. 1 2 3 "Predicting the present with bayesian structural time series" (PDF).
  7. ↑ "Bayesian variable selection for nowcasting economic time series" (PDF).
  8. ↑ "Inferring causal impact using Bayesian structural time-series models".
  9. ↑ "spikeslab" (PDF).
  10. ↑ "spikeSlabGAM" (PDF).
  11. ↑ "bsts" (PDF).
This article is issued from Wikipedia - version of the Thursday, April 21, 2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.