When consulted about what statistical analyses to use, or to pay attention to, I almost always deliver the ‘keep it simple smarty’ message.
Why?
1. Because more people will be able to understand (and isn’t that the point?)
2. Because everyone will be able to more quickly understand (and attention spans are short!)
3. Because usually simple describes or explains the data just as well as more complicated
4. Over complicating statistical methods can lend themselves to misleading or mis-interpreted results
My current favorite example is a very simple one — In higher ed I see a lot of use of means and standard deviations in reporting of data from scales, such as those in assessment rubrics (e.g. student work exceeds, meets, approaching, or is below standard). With this type of data advocate for looking first at frequencies or percentages. It may seem like the mean is simpler, because it’s one number versus several, but it’s much more difficult to interpret in many cases. Here’s an example:
Fake results from fake student writing assessment (n = 12)
Version 1: Applies numbers to the example rubric — exceeds = 4, meets = 3, approaching = 2, below = 1
Grammar: mean score 3.3 (standard deviation .75)
Sentence Structure: mean score 2.8 (standard deviation .83)
Version 2:
|
Grammar |
Sentence Structure |
|
|
below |
0% |
0% |
|
approaching |
17% |
42% |
|
meets |
42% |
33% |
|
exceeds |
42% |
25% |
Which makes more immediate sense to you? Can you see the differences more easily via the percentages than via the means?*
*There are certainly statisticians who disagree with use of mean and standard deviation outright with these types of variables. While I don’t come down hard and fast there, I do think it’s important to consider what you’re trying to learn when choosing just what numbers to pay attention to. In most cases, I find colleagues are looking to see more nuanced changes over time, or what the range of scores looks like, which a mean and standard deviation can’t alone provide.
While this is a very simple example, I think the same goes for choices between more complicated statistical approaches. I’m not saying that we shouldn’t use advanced statistical methods, only that if a simple option would net sufficiently explanatory results, why over-complicate things?
What do you think? When is it worth choosing complicated over simple? How can we simplify our explanations of more complicated analyses to make them more palatable, while still accurate and authentic?
I’m all for it! Perhaps because I am not a statistician, I become concerned when reporting findings that I haven’t done more complex statistical analyses. It helps me to be reminded that most often, we just need to understand the phenomenon (the evaluand) to be able to make reasonable judgments and effective programatic decisions based on the analyses. And for this, we can often use what is at our fingertips – basic statistical analysis.
Kim, You touched on a great point — about statisticians disagreeing with the use of the mean and standard deviation altogether for this type of data. Absolutely. This dataset is ordinal, which means all we can do is calculate the counts/frequencies of how many students fell into each cateogory. We can only calculate means and standard deviations on interval/ration data.
And, aside from all the technical reasons to favor your Version #2, there are also the practical reasons you mentioned. Time and time again, the teachers, social workers, and other practitioners I work with get way more meaning from frequencies than from means. Means are meaningless in so many situations.
Great post.
P.S. Version #3 would be a chart, right? 🙂
I’d love to see how you might visualize this data.
Totally Ann 🙂
And yet I see means etc used frequently in higher ed. It was quite puzzling at first but I’ve gotten used to the idea that some folks feel that it can be appropriate (I’m still not there myself, but making the most of what I get at times).
I would probably do a simple column or bar chart, honestly. Maybe labeled with the number of students in each category (meets, etc) rather than these percentages to make sure it’s clear we’re talking about small numbers here, so long as the n is consistent (that is often the case for us as a relatively small school). Maybe small multiples if there were actually more categories being assessed (as there usually are).
Though, now that you’ve got me thinking about it, I certainly could also use the nifty aligned stacked bar chart that you and Stephanie have written about…. Would depend on the planned use for the visualization I think. Do folks need a quick look at each area individually or would a composite chart with all the areas as aligned stacked bars be digestible and easier to share (depends on how it’s being shared I think)…. Thanks for asking!!