In
a previous post I applied a Bayesian ordered-probit model to movie ratings and showed how the results differ from treating the data as if they were metric. The metric model used frequentist
t tests (because that's what most applied researchers would do). In this post, I re-analyze that data as if they metric but using a Bayesian model that has the same hierarchical structure as the Bayesian hierarchical ordered-probit model I used before. Here we can compare ordered-probit to metric treatments with all else held constant. Spoiler: Same conclusions, of course. The ordered-probit model fits the data much better than the metric model. Don't treat ordinal data as metric.
Please see the
previous post for details about the data and the models. In particular, note that there is hierarchical structure on the standard deviations across movies, but not on the means across movies. In other words, there is no direct shrinkage on the means.
Here is a repeat of the data, from 30 movies, fit by the ordered-probit model:
The pink bars (above) are frequency histograms of the data from each movie; the blue dots (with vertical blue whiskers) are the posterior predictions and 95% HDIs for the ordered-probit model. A very good fit, all in all.
Here is some news. The data fit by the metric model. Here a 1-star rating is considered to be a score of 1.0 on a metric scale, a 2-star rating is considered to be a score of 2.0 on a metric scale, and so on. Each histogram is fit by a normal distribution (as is assumed by t tests, ANOVA, etc.). The result:
The superimposed blue curves (above) are a smattering from the posterior distribution. Not a very good fit to the data distributions.
More news. The means of the ordered-probit model plotted against the means of the metric model, with 95% HDI's:
You can see (above) that the rank ordering of the movies is quite different for the two models!
Case in point (and more news):
You can see (above) that the ordered-probit puts movie 10 well above movie 26, but treating the data as metric yields the opposite conclusion. Which conclusion is more appropriate? Clearly the ordered-probit describes the data much more accurately than treating the data as metric. Watch movie 10 before movie 26.
And don't be a free rider on the rating system. If you use the rating system, give a rating.
I am currently helping a colleague make sense of her data. Both the outcome and covariate of main interest are Likert scales. Presumably the correct way of going about the analysis would be to first retrieve the underlying continuous construct for each variable, for each individual and then proceed with a standard linear regression between those inferred measures?
ReplyDeleteSorry if this is an obvious question but this is the first time I'm working (appropriately!) with this type of data. Really glad I found your preprint before mucking this one up.
When the independent variable is ordinal, you have various options. Here are a few. One option is to treat the levels of the ordinal independent variable as categories of a nominal scale. That is, treat the IV as nominal (analogous to an ANOVA). That way you're predicting the DV for each level of the nominal IV and can do comparisons of levels of the IV, but you do not have any trend line across levels. As another option, you can pretend the levels of the ordinal IV are at estimated values on a continuous scale: Fix the first and last levels, and estimate the interior levels that best accommodate the trend you're assuming. Those two options assume you're doing regression of a DV on an IV. But another option is to treat both variables as DV's in a multivariate model. Then you're doing a sort of factor analysis, with a latent scale describing both ordinal values simultaneously.
Delete