Now you may have seen a headline recently talking about skyrocketing mental health problems among students.  Specifically, this one from the Toronto Star, which says, among other things:

A major survey of 25,164 Ontario university students by the American College Health Association showed that between 2013 and 2016, there was a 50-per-cent increase in anxiety, a 47-per-cent increase in depression and an 86-per-cent increase in substance abuse. Suicide attempts also rose 47 per cent during that period.

That’s a pretty stunning set of numbers.  What to make of them?

Part of what’s going on here is looking at the size of the increase instead of the size of the base.  If the incidence of something goes from 1% to 2% in the population, that can be accurately expressed either as “a one percentage point increase” or “IT DOUBLED!”.   The increase for the numbers on “attempted suicide in the last 12 months”, for instance, rose from 1.3% to 2.1%.  With such a tiny base, double-digit increases aren’t difficult to manufacture.

(in case you’re wondering whether these figures are a function of possible measurement error, the answer is no.  With a 40,000 student sample, the margin of error for an event that happens 1% of the time is 0.1, so a jump from 0.8% is well beyond the margin of error).

Now, the Star is correct, there is a very troubling pattern here – across all mental health issues, the results for 2016 are significantly worse than for 2013 and troublingly so.  But it’s still a mistake to rely on these figures as hard evidence for something major having changed.  As I dug into the change in figures between 2013 and 2016, I was amazed to see that in fact figures were not just worse for mental health issues, but for health and safety issues across the board.  Some examples:

• In 2013, 53.4% of students said their health was very good or excellent, compared to just 45.3% three years later
• The percentage of students whose Body Mass Index put them in the category of Class II Obese or higher rose from 3.15 to 4.3%, a roughly 35% increase.
• The percentage of students with diabetes rose by nearly 40%, migraine headaches by 20%, ADHD by nearly 25%
• Even things like incidence of using helmets when on a bicycle or motorcycle are down by a couple of percentage points each, while the percent saying they had faced trauma from the death or illness of a family member rose from 21% to 24%.

Now, when I see numbers like this, I start wondering if maybe part of the issue is an inconsistent sample base.   And, as it turns out, this is true.  Between 2013 and 2016, the institutional sample grew from 30 to 41, and the influx of new institutions changed the sample considerably.  The students surveyed in 2016 were far more likely to be urban, and less likely to have been white or straight.  They were also less likely to have been on varsity teams or fraternity/sorority members (and I suspect that last one tells you something about socio-economic background as well, but that’s perhaps an argument for another day).

We can’t tell for certain how much of the change in reported health outcomes have to do with the change in sample.  It would be interesting and helpful if someone could recalculate the 2016 data using only data from institutions which were present in the 2013 sample.  That would provide a much better baseline for looking at change over time.  But what we can say is that this isn’t a fully apples-to-apples comparison and we need to treat with caution claims that certain conditions are spreading in the student population.

To conclude, I don’t want to make this seem like a slam against the AHCA survey.  It’s great.  But it’s a snapshot of a consortium at a particular moment in time, and you have to be careful about using that data to create a time series.  It can be done – here’s an example of how I’ve done it with Canadian Undergraduate Survey Consortium data, which suffers from the same drawback.  Nor do I want to suggest that mental health isn’t an issue to worry about.  It’s clearly something which creates a lot of demand for services and the need to be met somehow (though whether this reflects a change in underlying conditions or a change in willingness to self-identify and seek help is unresolved and to some degree unresolvable).

Just, you know, be careful with the data.  It’s not always as straightforward as it looks.