tag:blogger.com,1999:blog-3240271627873788873.post666968428952414389..comments2024-03-26T06:46:11.752-04:00Comments on Doing Bayesian Data Analysis: Stopping and testing intentions in p valuesJohn K. Kruschkehttp://www.blogger.com/profile/17323153789716653784noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-3240271627873788873.post-70063491952490891792014-12-26T21:22:56.538-05:002014-12-26T21:22:56.538-05:00Whenever you generate a sampling distribution, you...Whenever you generate a sampling distribution, you must specify what constitutes a sample. Therefore you must specify if the sample has fixed N, fixed duration (with random N), or some other stopping rule. And you must specify which sample statistics you are sampling, which means which tests you are conducting. The sampling distribution depends on the intended stopping criterion and testing intentions. In the Bayesian approach there are NO sampling distributions. The distributions are of credible <i>parameters values</i> given the single fixed set of observed data. John K. Kruschkehttps://www.blogger.com/profile/17323153789716653784noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-73251783802840947372014-12-26T19:52:28.058-05:002014-12-26T19:52:28.058-05:00But why do p-values and stopping intentions come i...But why do p-values and stopping intentions come into this at all? You can bootstrap to find the sampling distribution of either mean, or the difference in means. Are these not the same as the distributions estimated by your model?Unknownhttps://www.blogger.com/profile/16031352087257980585noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-86280602148989963602014-12-26T16:42:44.542-05:002014-12-26T16:42:44.542-05:00Bootstrapping (a.k.a. resampling) is all about gen...Bootstrapping (a.k.a. resampling) is all about generating a sampling distribution, just like traditional <i>t</i> tests. The only difference is that we don't sample from a hypothetical normal population, we resample from the data (which stand as the best available representation of the population). But the bootstrapped sampling distribution still depends on the stopping and testing intentions like any other <i>p</i> value or confidence interval.John K. Kruschkehttps://www.blogger.com/profile/17323153789716653784noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-28552575101507010592014-12-26T15:18:25.712-05:002014-12-26T15:18:25.712-05:00Very nice work. I'm glad to see people re-exam...Very nice work. I'm glad to see people re-examining the fundamentals, and the t-test always seemed suspect to me.<br /><br />Question: could we do equivalent estimation with a bootstrapped approach, to generate a sampling distribution of difference of means? It seems to me this distribution has the same information as the distribution of the difference of mean parameters that you derive from your model. But bootstrapping and makes fewer assumptions and is both conceptually and practically simpler.Unknownhttps://www.blogger.com/profile/16031352087257980585noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-71358470115675314892014-08-12T21:36:25.666-04:002014-08-12T21:36:25.666-04:00Dear Dr. Mayo: Thank you for your comment. The vid...Dear Dr. Mayo: Thank you for your comment. The videos linked in the blog post are all about NOT ignoring the increased probability of erroneously rejecting or accepting the null with sequential testing. The videos show that Bayesian decision rules --including highest density interval with ROPE-- result in increased error rates that asymptote below 100%, unlike conventional NHST which rises to 100% false alarm rates with sequential testing. The videos show that any stopping decision based on accept/reject produces bias in the estimate (conditional on the decision). The videos argue that a better stopping rule than accept/reject is achieving precision. (This is what is done, for example, in political polling.) Importantly, precision is better measured by a Bayesian posterior distribution than by a frequentist confidence interval. --JohnJohn K. Kruschkehttps://www.blogger.com/profile/17323153789716653784noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-65474413739751609472014-08-12T18:53:43.777-04:002014-08-12T18:53:43.777-04:00It's rather disturbing, particular in this day...It's rather disturbing, particular in this day and age, that the Bayesian is prepared to countenance stopping til the data support a hypothesized real effect, ignoring the increased probability of erroneously doing so. In standard normal testing, for instance, one is guaranteed to erroneously reject (or equivalently erroneously exclude the true value from a HPD region) with maximal probability. Cherry-picking, multiple testing, post-data choices of subgroups---all these gambits are touted as the "simplicity and freedom" (Savage) offered by the Bayesian Way. Their reference priors cannot save the day. True one may find, at times, a skeptical prior to restore what the error statistician achieves directly (and with a clear rationale). But if these gambits are not to be frowned upon for destroying error control, on what basis can one demand a sufficiently skeptical prior?MAYO:ERRORSTAThttps://www.blogger.com/profile/02967648219914411407noreply@blogger.com