Trade-off of between-group and within-group variance (and implosive shrinkage)

Background: Consider data that would traditionally be analyzed as single-factor ANOVA; that is, a continuous metric predicted variable, \(y\), and a nominal predictor, "Group." In particular, consider the data plotted as red dots here:

A Bayesian approach easily allows a hierarchical model in which both the within-group and between-group variance are estimated. The hierarchical structure imposes shrinkage on the estimated group means. All of this explained in Chapter 19 of DBDA2E. 

The data above come from Exercise 19.1, which is designed to illustrate "implosive shrinkage." Because there are only a few data points in each group, and there are lots of groups with little variance from baseline, a reasonable description of the data merely collapses the group means close to baseline while expanding the estimate of within-group variance. 

The purpose of the present post is to show the trade-off in the posterior distribution of the estimated within-group variance and between-group variance, while also providing another view of implosive shrinkage.

After running the data through the generic hierarchical model in Exercise 19.1, we can look at the posterior distribution of the within-group and between-group standard deviations. The code below produces the following plot:

mcmcMat = as.matrix( mcmcCoda )
openGraph()
plot( mcmcMat[,"aSigma"] , mcmcMat[,"ySigma"] , 
      xlab="Between-Group Sigma" , ylab="Within-Group Sigma" , 
      col="skyblue" , cex.lab=1.5 , 
      main="Trade-Off of Variances (and implosive shrinkage)" )
abline( lm( mcmcMat[,"ySigma"] ~ mcmcMat[,"aSigma"] ) )

Notice in the plot that as the between-group standard deviation gets larger, the within-group standard deviation gets smaller. Notice also the implosive shrinkage: The estimate of between-group standard deviation is "piled up" against zero.

A foolish prior on consistency is the howling of shrunken minds

(Photo from Professor Sylvain Fiset's web site.)

Professor Sylvain Fiset has been using Bayesian methods (from DBDA2E)  to analyze short-term memory of dogs. In each experimental trial, a dog watched a human experimenter tuck away a treat in one of several boxes. After a moment, the dog was allowed to retrieve the treat. Each trial was scored correct if the dog went directly to the correct box and scored wrong otherwise. (Actually, the task was a little harder, involving two hidden treats, but that detail isn't crucial here.) Thus, each dog had \(z_s\) correct out of \(N_s\) trials. We would like to estimate the underlying ability of each dog, \(\theta_s\), and the typical ability of the group of dogs, \(\omega\), and the consistency across the dogs, \(\kappa\). This is exactly the structure of the therapeutic touch example in Chapter 9 of DBDA2E. So, it should be trivial to simply read in the data, run the pre-existing script, and bark in joy at the output!

But, when Sylvain did that, the MCMC chains would not converge. Howls! He asked me what was going on, and I determined that the data showed a fairly large variability across dogs, and for whatever reason the canned prior on the consistency parameter, \(\kappa\), was allowing the MCMC chain to "get stuck" at very low values of \(\kappa\). So I recommended tweaking the prior on \(\kappa\) a modest amount to keep it away from zero while still being broad and noncommittal. It then worked fine. Barks of joy!

However, only the next day (i.e., today), I realized that Past Self had already thought about implications of the prior on \(\kappa\). (If only my human long-term memory were better!) Past Self was so concerned with this issue that he wrote Exercises about it in DBDA2E! See Exercises 9.1 and 9.2, with solutions available at the book's web site. Here's the idea: Notice that the consistency parameter, \(\kappa\), constrains how far apart the individual ability parameters, \(\theta_s\), will spread apart. The higher the consistency, the more shrinkage on the individual estimates. If the prior on \(\kappa\) avoids small consistencies, it implies that individual parameters will be relatively near each other. In other words, even if the prior on \(\kappa\) seems broad and noncommittal, it may imply stronger shrinkage on the individual estimates that you may intuit. Therefore, like Past Self, Present Self recommends that you take a look at the implied prior on the differences of individual values, \(\theta_s - \theta_t\), to check that the prior is appropriate for your application. Usually (for typical data sets with moderately large amounts of data) the shrinkage will be appropriate, but it's worth checking and thinking about.

By the way, I met Sylvain at Stats Camp, where I taught one of my workshops in Bayesian Methods. 

Finally, the title of this post is supposed to be directed at myself for having forgotten about Past Self's consideration of this issue, but in defense of Present Self the exercises were not about MCMC convergence. The title is a variation of a phrase from Ralph Waldo Emerson.

Doggerel Doggies: The Generalized Linear Model

The opening "poem" from Chapter 15 of DBDA2E. The color coding explains the correspondence of metaphor to math.