tag:blogger.com,1999:blog-3240271627873788873.post1843588582283789131..comments2024-03-26T06:46:11.752-04:00Comments on Doing Bayesian Data Analysis: How much of a Bayesian posterior distribution falls inside a region of practical equivalence (ROPE)John K. Kruschkehttp://www.blogger.com/profile/17323153789716653784noreply@blogger.comBlogger18125tag:blogger.com,1999:blog-3240271627873788873.post-8625644485070806392019-07-28T06:53:55.349-04:002019-07-28T06:53:55.349-04:00Thanks so much for the quick reply John. I just re...Thanks so much for the quick reply John. I just reread that section and it seems to suggest that if my chains have converged and are well mixed, Gelman–Rubin statistics are below 1.1 and ESS > 10,000, then there shouldn't be any great concern with interpreting HDIs limits near the ROPE limits (although I'm still not sure how to interpret the "very near" of p. 339...?) Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-58292876152486179522019-07-27T08:29:31.509-04:002019-07-27T08:29:31.509-04:00This article has more details about how to constru...This article has more details about how to construct a ROPE:<br />Kruschke, J. K. (2018). Rejecting or Accepting Parameter Values in Bayesian Estimation, Advances in Methods and Practices in Psychological Science, 1, 270-280. <br />The article is at https://journals.sagepub.com/doi/pdf/10.1177/2515245918771304<br />and important supplemental materials are at https://osf.io/fchdr/John K. Kruschkehttps://www.blogger.com/profile/17323153789716653784noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-10753560780652296122019-07-27T08:13:16.304-04:002019-07-27T08:13:16.304-04:00Hi Dylan. The answer to your question is discussed...Hi Dylan. The answer to your question is discussed extensively in Ch. 7 (Section 7.5) of the book (DBDA, 2nd Edition). Basic recommendation: For reasonably stable HDI limits, make the effective sample size (ESS), of whatever it is you're looking at, at least 10,000. John K. Kruschkehttps://www.blogger.com/profile/17323153789716653784noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-87859652871254572982019-07-27T01:48:59.651-04:002019-07-27T01:48:59.651-04:00Dear John,
I'm quite new to Bayesian statisti...Dear John,<br /><br />I'm quite new to Bayesian statistics but really appreciate the clarity of your explanations. In Chapter 12.1.1 (p. 339) of your book (DBDAE2), you write:<br /><br />"Thus, if the MCMC HDI limit is very near the ROPE limit, be cautious in your interpretation because the HDI limit has instability due to MCMC randomness. Analytically derived HDI limits do not suffer this problem, of course."<br /><br />Could you please give some advice on just how near is "very near" (e.g. a certain proportion of the HDI length?), and what we need to consider when interpreting such cases?<br /><br />Many thanks :)<br />DylanAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-46107287686564580472018-01-02T15:38:16.615-05:002018-01-02T15:38:16.615-05:00I am a new fans here using bayesian method. I am u...I am a new fans here using bayesian method. I am using 3 level hierachical log-linear growth model (non-centered slope & intercept) to estimate growth rate in plants growing in different pesticide concentrations. However, with the log-linear model, the difference of growth rate (slope) is extremely small.<br /><br /><br />I have the control compared with other 5 concentrations of pesticide. Mean difference of growth rates are described as below:-<br />comparison of mode of mean difference 95% HDI limit<br />concentrations (growth rate) farthers from 0<br />i) control vs 1mg/L pesticide 0.00576 -0.00433-0.02<br />ii) control vs 1.8mg/L pesticide -0.0153 -0.0256-0.0042<br />iii) control vs 3.2mg/L pesticide -0.0161 -0.0276-0.00505<br />iv) control vs 5.8mg/L pesticide 0.0483 0.0367-0.0582<br />v) control vs 10mg/L pesticide 0.0875 0.0777-0.098<br /><br />If using the suggested standard ROPE around zero, from -0.1 to +0.1 as above mentioned, there will be no effect of pesticide for all concentrations (whereby, it is not the case). So, seems with log-linear model, the ROPEvalue need to be adjusted. I am looking at 0.02 instead of 0.1. Does 0.02 sounds akward? Is there other forum I could post more information for the question? I have plotted the same plot as shown above, wish I could show it here. This is only result from one lab, there are other data from another 30 labs which has the same problem. I'm hopping if you could shed some light on the problem mentioned.Anonymoushttps://www.blogger.com/profile/03934609071603867935noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-897938707617448852017-11-15T08:16:52.528-05:002017-11-15T08:16:52.528-05:00A key feature of the ROPE+HDI approach is to be ab...A key feature of the ROPE+HDI approach is to be able to "accept" a parameter value (for practical purposes). In spirit it is much like frequentist equivalence testing. <br /><br />See this blog post about assessing null values:<br />http://doingbayesiandataanalysis.blogspot.com/2016/12/bayesian-assessment-of-null-values.html<br /><br />See this blog post about equivalence testing:<br />http://doingbayesiandataanalysis.blogspot.com/2017/02/equivalence-testing-two-one-sided-test.htmlJohn K. Kruschkehttps://www.blogger.com/profile/17323153789716653784noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-70995386100504560422017-11-15T07:54:47.827-05:002017-11-15T07:54:47.827-05:00Dear John,
the decision rules that you mentioned ...Dear John,<br /><br />the decision rules that you mentioned are: accept the null-hypothesis if the HDI is a subset of the ROPE and reject it if HDI and ROPE are non-overlapping. But what happens if they only partially overlap? Do you think it would be meaningful to condense these three cases into two cases, like in frequentist testing: (i) null-hypothesis can be rejected and (ii) null-hypothesis cannot be rejected (intersection of HDI and ROPE is nonempty)? In words, the null-hypothesis could then *not* be rejected if the 95% most credible values do *not* fall outside of the ROPE. My final goal is to compare several means and derive a letter coding, similar in spirit to Fisher's LSD, to indicate which pairs are not "significantly different".<br /><br />Thank you,<br />DominikAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-55394426425245348082017-04-03T08:40:35.327-04:002017-04-03T08:40:35.327-04:00The ROPE should be defined by considering what it ...The ROPE should be defined by considering what it means to be a negligible effect size, what it means for an effect to be "practically equivalent" to the landmark value. That's separate from whatever data you happen to have in the current study.John K. Kruschkehttps://www.blogger.com/profile/17323153789716653784noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-44936200948464869032017-03-30T15:40:33.941-04:002017-03-30T15:40:33.941-04:00Do you think that it would be informative if you d...Do you think that it would be informative if you defined the ROPE as the 95% HDI of the same data randomized? You could include it as another possibly interesting way point in your plot of the proportion of the posterior inside ROPE as a function of ROPE width. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-54724801995442066082016-03-18T15:36:01.728-04:002016-03-18T15:36:01.728-04:00Dear Anonymous March 18:
I think that question is...Dear Anonymous March 18:<br /><br />I think that question is already answered (perhaps implicitly) in my last comment. I like a decision rule that treats the densities seriously. To illustrate with an extreme contrived example, suppose that 98% of the distribution is outside the ROPE, but it is so spread out that its density is actually lower than the 2% of the distribution inside the ROPE. Then the most credible (highest density) values are inside the ROPE, even if most of the distribution is outside the ROPE.<br /><br />That's why I like a decision rule that actually uses the highest-density values. This isn't a uniquely correct decision rule, it's just one that makes a lot of sense if you treat densities seriously.John K. Kruschkehttps://www.blogger.com/profile/17323153789716653784noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-62959411114130064972016-03-18T13:43:13.568-04:002016-03-18T13:43:13.568-04:00Somewhat related to the loss function question, wh...Somewhat related to the loss function question, why use the HDI and not just compute P(\theta outside ROPE)? If it is "high" it seems like reason to reject the ROPE. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-55124157243291200242015-11-16T14:49:07.906-05:002015-11-16T14:49:07.906-05:00> 1. Which do you think more appropriate as a d...> 1. Which do you think more appropriate as a decision criterion a) require the HDI interval to be within ROPE, or b) set a lower limit on the posterior probability contained within ROPE.<br /><br />I take the posterior density seriously, which is why it makes sense to me to consider the relation of the HDI to the ROPE. The decision is to "accept" the null value is based on the 95% <i>most probable</i> values all being practically equivalent to the null value. The decision to "reject" he null value is based on the 95% <i>most probable</i> values all not being practically equivalent to the null value.<br /><br />Suppose instead that we accept a ROPEd value if (say) 95% of the posterior mass falls within the ROPE, and we reject a ROPEd value if (say) 95% of the posterior mass falls outside the ROPE. Then we would get weird conclusions when distributions are strongly skewed. Consider Figure 12.2, p.342, of DBDA2E, which shows a skewed distribution with its 95% HDI and its 95% equal-tailed interval (ETI). Suppose the ROPE boundaries are at the ETI boundaries. Then we would accept the ROPEd value despite the fact that there are parameter values with high posterior probability that are not practically equivalent to the ROPEd value. Or, suppose that the right side of the ROPE is at the left edge of the ETI in Figure 12.2. Then we would reject the ROPEd value despite the fact that there are parameter values with high posterior probability that are practically equivalent to the ROPEd value.<br /><br />> 2. What about a confidence level? For instance consider the philosophy behind a Bayesian fixed-in-advance beta-content tolerance interval (e.g., see Wolfinger) in which both a content (proportion) and a confidence level are specified. Specifying a confidence level provides a level of assurance that the result is repeatable. I think this question may apply mostly to a posterior predictive situation.<br /><br />I don't have much to say about this. I think, though, that you're leaning toward a frequentist criterion applied to a Bayesian posterior distribution. There's nothing inherently wrong with that, but it's a different animal than what's being pursued with the HDI and ROPE. With a frequentist criterion you need to make assumptions about your sampling intention --which you can do-- but then the criterion is linked to your specific sampling assumptions.<br />John K. Kruschkehttps://www.blogger.com/profile/17323153789716653784noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-29921587959141742612015-11-15T15:23:45.044-05:002015-11-15T15:23:45.044-05:002 questions on ROPE:
1. Which do you think more a...2 questions on ROPE:<br /><br />1. Which do you think more appropriate as a decision criterion a) require the HDI interval to be within ROPE, or b) set a lower limit on the posterior probability contained within ROPE.<br /><br />2. What about a confidence level? For instance consider the philosophy behind a Bayesian fixed-in-advance beta-content tolerance interval (e.g., see Wolfinger) in which both a content (proportion) and a confidence level are specified. Specifying a confidence level provides a level of assurance that the result is repeatable. I think this question may apply mostly to a posterior predictive situation.<br /><br />Thank you for your thoughts!!<br /><br />Dave LeBlond (david.leblond@sbcglobal.net)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-55569595086842207962014-06-11T23:01:42.598-04:002014-06-11T23:01:42.598-04:00Is there a way to frame this criterion as the solu...Is there a way to frame this criterion as the solution to the minimiation of the posterior expected loss for some loss function? If so, what would be the corresponding loss function?<br /><br />I started a question here: http://stats.stackexchange.com/questions/103067/rope-vs-hdi-loss-function and it would be great to get your input or someone elses on it.<br /><br />Thanks for the great post!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-5651811995274197432013-08-20T17:05:41.191-04:002013-08-20T17:05:41.191-04:00Mike:
That's a good point you make about the ...Mike:<br /><br />That's a good point you make about the meaning of the farthest HDI limit. <br /><br />And thank you again for creating the BEST package!<br /><br />--JohnJohn K. Kruschkehttps://www.blogger.com/profile/17323153789716653784noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-12097552537394674102013-08-11T06:32:55.108-04:002013-08-11T06:32:55.108-04:00Very nice post and useful function. A version of B...Very nice post and useful function. A version of BEST package with it is now on GitHub and I'll submit to CRAN after I've run some more checks.<br /><br />BTW is it clear that the landmark "HDI limit farthest" is the radius at which the HDI lies entirely within the ROPE? Ie. your preferred decision rule for accepting the hypothesis. Mike Meredithnoreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-19067670677651698302013-08-09T13:15:18.228-04:002013-08-09T13:15:18.228-04:00I like your string of thought, and it might lead t...I like your string of thought, and it might lead to a long thread of loosely tied responses. Wherever these PUNishable comments take us, let's agree in advance not to noose our sense of humor!John K. Kruschkehttps://www.blogger.com/profile/17323153789716653784noreply@blogger.comtag:blogger.com,1999:blog-3240271627873788873.post-84724960826516248172013-08-09T05:47:45.738-04:002013-08-09T05:47:45.738-04:00Interesting to read more about ROPE! It does feel ...Interesting to read more about ROPE! It does feel a little bit like a Bayesian twist on hypothesis testing. A large ROPE results in a higher chance of accepting the null while a tight ROPE (tightrope?) gives a higher chance of rejecting the null. It would be interesting to see a paper where the ROPE decision rule is compared to other possible decision rules (Neyman–Pearson, Bayes factor, etc.).<br /><br />By the way, since you call it a ROPE, would it be ok to introduce some extra rope related nomenclature? For example, when the ROPE null can't be rejected nor accepted this could be a "tie", a reasonably tight ROPE could be a "tightrope", an unreasonable tight ROPE could be a "knot", etc.Rasmus Bååthhttps://www.blogger.com/profile/16575386339856902265noreply@blogger.com