Wednesday, February 25, 2015

"Is the call to abandon p-values the red herring of the replicability crisis?"


In an opinion article [here] titled "Is the call to abandon p-values the red herring of the replicability crisis?", Victoria Savalei and Elizabeth Dunn concluded, "at present we lack empirical evidence that encouraging researchers to abandon p-values will fundamentally change the credibility and replicability of psychological research in practice. In the face of crisis, researchers should return to their core, shared value by demanding rigorous empirical evidence before instituting major changes."

I posted a comment which said in part, "people have been promoting a transition away from null hypothesis significance testing to Bayesian methods for decades, long before the recent replicability crisis made headlines. The main reasons to switch to Bayesian have little directly to do with the replicability crisis." Moreover, "It is important for readers not to think that Bayesian analysis merely amounts to using Bayes factors for hypothesis testing instead of using p values for hypothesis testing. In fact, the larger part of Bayesian analysis is a rich framework for estimating the magnitudes of parameters (such as effect size) and their uncertainties. Bayesian methods are also rich tools for meta-analysis and cumulative analysis. Therefore, Bayesian methods achieve all the goals of the New Statistics (Cumming, 2014) but without using p values and confidence intervals."


See the full article and comment at the link above.


Monday, February 23, 2015

Journal bans null hypothesis significance tests

In a recent editorial [here], the journal Basic and Applied Social Psychology has banned the null hypothesis significance testing procedure (NHSTP). "We hope and anticipate that banning the NHSTP will have the effect of increasing the quality of submitted manuscripts by liberating authors from the stultified structure of NHSTP thinking thereby eliminating an important obstacle to creative thinking. The NHSTP has dominated psychology for decades; we hope that by instituting the first NHSTP ban, we demonstrate that psychology does not need the crutch of the NHSTP, and that other journals follow suit."

In a short bit about Bayesian analysis, the editorial says, "The usual problem with Bayesian procedures is that they depend on some sort of Laplacian assumption to generate numbers where none exist." I think here the editors are too focused on Bayesian hypothesis tests instead of on the much broader application of Bayesian methods to parameter estimation. For example, in the 750 pages of DBDA2E, I never mention the Laplacian assumption because the procedures do not depend on it. Despite their narrow view of Bayesian methods, I am encouraged by the bold move that might help dislodge NHST.

Sunday, February 8, 2015

I've got variable Y that I want to predict from variables X1, X2, etc. What should I do?


For questions like yours -- I've got variable Y that I want to predict from variables X1, X2, etc.; What should I do? -- the best answer is usually informed by background knowledge of the domain. Generic models, like multiple linear regression, don't always make the most meaningful answer.

For example, suppose you're trying to predict the amount of fencing (Y) you'll need for rectangular lots of length X1 and width X2. Then a linear regression would serve you well. Why? Because we know (from background knowledge) that perimeter is a linear function of length and width.

But suppose you're trying to predict how much grass seed you'll need for the same lot. Then you'd want a model that includes the multiplicative product of X1 and X2, because that provides the area of the lot.

As another example, suppose you're trying to predict the installed length of a piece of pipe (Y) as a function of the date (X). You know that pipe expands and contracts as some function of temperature. And you also know that temperature cycles sinusoidally (across the seasons of a year) as a function of date. So, to predict pipe length as function of date, you'd use some trend that incorporates the expansion function on top of a sinusoidal function of date.

Whatever model you end up wanting, it can probably be implemented in JAGS (or BUGS or Stan). That's one of the beauties of the Bayesian approach with its general purpose MCMC software.