Periodically, in Canada, someone comes up with a statistic about higher education.  Doesn’t matter if it makes the least bit of sense – as long as it serves somebody’s political narrative.  This statistic can go ON and ON unchallenged for years unless someone steps on it quickly (and I should know: I came up with a doozy about a decade ago).

Yesterday, one such statistic popped up and it’s so juicy you just know it’s going to used constantly even though it is TERRIBLE.  You may have seen the Globe article about it: one-in-four recent STEM grads from Waterloo, UBC and Toronto are working outside Canada, and this is being touted as evidence of a national “brain drain”.  (If this reminds you of something, remember the claims made by the good folks at the Canadian Council of Innovators a couple of weeks ago?  You know those guys are going to be all over this – and in fact their think piece was quite clearly based around an early draft of this paper).

Well, now.  What to make of this?

Let’s start by looking at the sample involved.  The authors of the piece – Zach Spicer of the University of Toronto and Nathan Olmstead and Nicole Goodman of Brock University (read the whole thing here) chose three of the country’s top STEM universities.  Given that the question they are trying to answer is “where are recent Canadian STEM grads seeking employment?” it might have been useful to consider whether or not those three programs are representative of the country as a whole or whether these institutions might have above-average migration rates (the fact that they are home to disproportionately large numbers of international STEM students who wouldn’t necessarily be expected to stay in Canada after graduation seems to me to be a pretty obvious factor to consider).  If the authors did consider this question – and given the massive disparities in the results between institutions (see below), you’d think it would have occurred to them – it is not evident from the report.

Now, look at the actual way the data was collected.  Over the years 2015 and 2016, those three institutions graduated nearly 30,000 STEM students across 34 STEM programs.  To quote the report: “To make the data collection more manageable we developed a sample of 22 programs, which produced 6,603 graduates”.  Got that?  They excluded the programs which in fact produced 78% of the total graduates of these institutions.

Why were these specific 12 excluded?  Is there perhaps something special about the 22 they chose which might make them more or less likely to move abroad?  We don’t know – the authors don’t say.  But if you look carefully, you realise that the over half the sample is from Waterloo, and the Waterloo sample is effectively only Math, Computer Science and the IT end of Engineering (no civil, no mechanical).  Chemistry and Physics were included, but they represent only about 3% of the Waterloo sample.  There are similar biases at the other two institutions.  So let’s be honest – this sample isn’t about STEM, it’s specifically about fields that matter to the software/IT industries.  Why the authors chose to make larger claims about STEM instead of limiting it to the IT industry is a bit mystifying.

Next, we come to the actual sample obtained.  Of those 6,603, they managed to track down 3,162 students based on their LinkedIn profiles.  Now, I am not one to blindly condemn people for innovative methodology.  Tracing graduates is hard to do and very expensive to do well.  So as an off-the-cuff attempt to try to get at the question of graduate migration, this is not bad.  The problem is when you try to pass off the results as meaningful without any acknowledgement that excluding anyone who doesn’t use LinkedIn might introduce a teeny, tiny amount of bias to the results.

[the_ad id=”12598″]

So what did this limited/biased sample find?  Well, apparently 30% of computer science and computer engineering graduates and 66% of software engineering graduates (of which there were 90 in the sample), left the country mostly to go to the United States.  But in the STEM programs which actually have the big national enrolment numbers – say, Mechanical Engineering, Chemistry or Biology, the numbers are all under 10% (indeed, for biology the number is exactly zero).  The only way you get to some dreadful number for STEM as a whole is by torqueing the sample.  Which is precisely what the authors did.  And even then, you have to gloss over some pretty huge inter-institutional differences.  In those specific IT-related disciplines, the percentage of graduates leaving the country is 41% at Waterloo, but only 19% at UBC.

Not only would I argue that this is not evidence of a national STEM brain drain, I don’t think I’d argue it is particularly strong evidence of an IT-related brain drain either.  Rather, I would suggest what this data shows is that the world’s top IT companies are hiring a hell of a lot of grads from one university (Waterloo) which is one of the world’s top IT universities.  This is at best fodder for a Waterloo marketing campaign, not evidence of some kind of national crisis.

Some credit to the authors here – unlike the CCI gong show a couple of weeks ago, they point out that IT salaries are substantially lower in Canada than in the US and indeed go on to quantify the gap quite nicely (Tech workers’ salaries in Toronto are between 50-66% of what they are in the US).  And with those kinds of gaps the correct question is not “why are we losing so many IT workers” it’s “how the hell are we keeping any at all”?

But not too much credit.  After all, this is a paper on graduate migration which does not control for international student status, which insinuates that three universities are a reasonable sample from which to extrapolate national trends, uses a sample frame which excludes 78% of the population on which it purports to report, and then of that group focuses on a biased 50% sample.  The claims made in the paper are much stronger than the data can credibly support.  Quote this study at your peril.