Better than the t test: Robust Bayesian Comparison of Groups
[An updated post appears HERE.] It's been said that if all you're doing is a t test, Bayesian methods don't get you anything more. Wrong! Here's a manuscript showing that robust Bayesian estimation (not Bayesian model comparison involving Bayes factors) produces far more information about the difference of means, the difference of standard deviations, the influence of outliers, and power of the test. The Bayesian method can also accept the null, not merely reject it. The software and programs are free and easy to use. It is time for robust Bayesian estimation to supersede traditional methods.
Hi John,
ReplyDeleteI agree with your general conclusions that your approach is more informative than a standard t-test. Though I would like to ask four questions:
- What would be your reply to someone stating that your comparison is not fair as your method allows to have outliers (because you are using a t-distribution for your data), but a normal t-test is specificcally for normally distributed data only and hence you are comparing apple and oranges?
- Why did you not specify how you simulated your examples in the article. E.g. were the data in the low sample size data set drawn from different or from the same distribution? Hence what in this case was the correct answer?
- Did you also try to allow for a different tdf for both groups and how this approach would compare to others.
- Finally, is there a limit on sample size (as you are estimating 5 parameters instead of a single parameter in a "normal" t.test. Though to be fair someone would need to include an F-test to check for the variances in a frequentist approach.), where you would not recommend the approach (say if groups do consist of less then five samples).
And finally did you also have a look how differences in samples sizes of the groups are affecting the conclusions, as this could be another potential advantageous as this is not taken into account in the standard t.test at all, I think.
In general I like the approach, and I would like to see how the anova equivalent would look like, as this is the type of data which is more relevant to me. I can see the extensions to more levels in a factor, but not to sure how to include interactions between factors if there is more than one factor
Regards, Bernd
In reply to Bernd's comment:
ReplyDeleteQ. What would be your reply to someone stating that your comparison is not fair as your method allows to have outliers (because you are using a t-distribution for your data), but a normal t-test is specificcally for normally distributed data only and hence you are comparing apple and oranges?
A. The robust Bayesian approach does apples AND oranges. That is, the robust approach handles cases with outliers and cases without outliers. Traditional t tests do not handle outliers. Is there a "robust" NHST t test, i.e., one that uses t distributions to describe the data? Let me know if there is. And even if there is, what information could it possibly yield? Only the impoverished NHST info, namely, a p value and a confidence interval based on p values. p values are ill-defined and confidence intervals have no distributional info.
Q. Why did you not specify how you simulated your examples in the article. E.g. were the data in the low sample size data set drawn from different or from the same distribution? Hence what in this case was the correct answer?
A. Nothing deceptive is going on, but you're right I should have explained. The data are just random samples from either t(df=2), to get samples with outliers, or from normal distributions, for samples without outliers. I linearly re-scale the random samples so that they have the desired sample mean and sd. The Bayesian posterior recovers the "true" values nicely.
Q. Did you also try to allow for a different tdf for both groups and how this approach would compare to others.
A. I used the same tdf for both groups because outliers tend to be few in number, and by using data from both groups to estimate tdf there can be a more stable estimate. But, it is merely an assumption that the two groups have the same underlying tdf. Fortunately, the Bayesian program can be easily modified to estimate two tdf's if an application strongly suggests it.
Q. Finally, is there a limit on sample size (as you are estimating 5 parameters instead of a single parameter in a "normal" t.test. Though to be fair someone would need to include an F-test to check for the variances in a frequentist approach.), where you would not recommend the approach (say if groups do consist of less then five samples).
A. Another reason to love the Bayesian approach is that it doesn't care what the samples sizes are. When you have very small sample sizes, the posterior simply reveals very broad, uncertain estimates. If you want, just for fun, you can enter the data one datum at a time and see what happens to the posterior distribution.
Q. And finally did you also have a look how differences in samples sizes of the groups are affecting the conclusions, as this could be another potential advantageous as this is not taken into account in the standard t.test at all, I think.
A. Bayesian analysis doesn't care about equal or unequal sample sizes, and it correctly shows greater uncertainty in the parameters of groups with smaller sample sizes. For example, if one group has sample size of N1=10 and the second group has sample size of N2=100, the marginal posteriors of mu1 and sigma1 will be much wider than the marginal posteriors of mu2 and sigma2. (I'm not sure that unequal sample sizes are a problem for the traditional t tests. Are they really?)
Q. In general I like the approach, and I would like to see how the anova equivalent would look like, as this is the type of data which is more relevant to me. I can see the extensions to more levels in a factor, but not to sure how to include interactions between factors if there is more than one factor.
A. The programs for two-factor ANOVA in the book are trivially changed to use a t distribution for the likelihood function instead of a normal distribution. In other words, it's simple to do "robust" Bayesian ANOVA, whether with one factor or more.
I'd found the paper via Google, but am glad I also found this blog posting for the Q&A section. I've recommended the blog, paper, and your book to someone on Cross Validated who asked, "How do Bayesians compare distributions?"
ReplyDeleteI have your book and I enjoy both the content and style. Still a bit puzzled by the dog theme on the cover, though! ;-)
Thanks!
Why the dogs? See:
ReplyDeletehttp://doingbayesiandataanalysis.blogspot.com/2012/02/why-puppies.html