Most university rankings (U-Multirank is the big exception) take a league table format originally used by esteemed psychologist, eugenicist and baseball enthusiast James McKeen Cattell in his early rankings early 20th century (for more on Cattell see back here). One effects of borrowing league tables as a metaphor is that there is an implicit assumption that the inhabitants of that table are able to move up and down the league table as baseball or football teams do. If a team can crash like the Orioles (1st place to godawful hopeless in 4 years), or rise, against-all-odds to the very top like Leicester City (Hi, Jim!), so too can institutions, right?
Well, no. And this is part of the problem. By using a league table format, rankings foster an illusion of mobility which is completely at odds with what we know about rankings which is that we barely ever see any movement at the top because big rich old universities have such built-in advantages that they can never be shifted from the top. Which is why baseball/football league tables are the wrong way to present data: a better metaphor can be found are sumo rankings known as banzuke.
For those of you who don’t have a sumo obsessed nine year-old daughter to explain the banzuke to you, allow me to expand on this a bit. There are around 700 sumo wrestlers, divided into six divisions. Only the 70 who occupy the top two divisions – Juryo and Makuuchi – are actually professional (the rest essentially fight for room, board and a shot at elevation to professional status). There are 6 tournaments a year lasting 15 days each. In the bottom four divisions, wrestlers fight 7 times a tournament; in the top two they fight every day. The banzuke – which is re-done at the start of each bashō (tournament) – works like an enormous squash ladder. Win more matches than you lose and you move up the banzkue; lose more than you win and you fall.
Now, this system holds until you get to the very summit of the sport. Makuuchi, the top division, has 40 members. Most of these hold the rank of “Maegashira” followed by a number – Maegashira 12, Maegashira 6, all the way up to Maegashira 1 (there are two of each, one “east” and one “west”, for reasons that are of no serious import). Above that are two levels of junior champions (Komusubi and Sekiwake), and then the very special ranks for which there are different sets of rules – champions (Ozeki) and grand champions (Yokozunas). Unlike everyone else, Ozeki do not get demoted if they post a losing record: it takes two consecutive losing records to get demoted. And Yokozunas can never be demoted: a grand champion is a grand champion for life (if they ever have a losing record, they actually have to retire, but this almost never happens).
The way a sumo tournament works is that you don’t get people of radically different ability paired against one another. If you’re a Maegashira 10, you’re going to spend the tournament paired against guys who are somewhere between Maegashira 14 and Maegashira 6. Yokozunas rarely get paired with anyone below Maegashira 4 (and even those bouts are usually ridiculous mismatches). Thus, although there is a single, really long 700-name banzuke, there is no illusion that it is a single competition. The top is the top. The middle is the middle. They are not comparable.
You can probably see where I am going with this. Canada has the “big three” schools – Toronto, McGill, UBC and internationally there is a kind of “big six” (Harvard, MIT, Princeton, Stanford, Oxford, Cambridge). These institutions are Yokozunas. Not only are they are not on the same plane as everyone else, they are in fact the yardstick by which everyone else is measures. Their money, size and profile give them what amounts to Yokozuna status. And good luck getting in the dohyō with them.
Below them there are another bunch of institutions which are not quite there, but clearly are ahead of most of the pack. In Canada, that would be Alberta, Montreal, McMaster and (maybe, depending on the ranking) Ottawa, Calgary as well – internationally, it would include places like Yale, UCLA, the University of Washington, and on a good day the University of Tokyo and maybe even Toronto. Again, good luck competing with these guys. They might care about how they measure up against each other, but the rest of the higher education universe is not really on their radar. And so on, down the line. Maybe the top 200 institutions globally or so are analogous to Makuuchi, the next 300 or so to Juryo.
But here’s the important thing: as you approach the top of the table, no one actually cares what your rank is within the division. By design, it’s pretty much impossible to maintain your exact position in the banzuke – you either move up or down each tournament. So take one favourite rikishi of mine, Chiyonokuni: he strated this year at Maegashira 7, then dropped to M10, M11 before bouncing all the way up to M2 after a good tournament before dropping down again to M4. He was pretty much hopeless in the tournament that ended Sunday and in November will probably be ranked somewhere around M10 again.
Here’s the key thing: no one thinks these gyrations actually mean anything. Everyone knows Chiyonokuni is a mid-table guy, whether he’s Maegashira 4 or Maegashira 10. No one gets excited if he rises or falls a few spots. Chyonokuni is University of Guelph, basically. He’s good, he’s solid but he’s never going to make Ozeki, and he is neither going to fall into Juryo any time soon. Parsing each tournament’s banzuke for any more information than that is ridiculous.
And that’s more or less the way we should read university rankings. Those at the top are impossible to dislodge. Tiny movements in the banzuke are mostly noise. Comparisons (matches) should be restricted to with people near you on the banzuke, comparisons with people in different divisions are not just meaningless but effectively impossible.
Yoi shumatsu o!
Tweet this post
I was brought to this article from the Tachiai blog (and I’m a college professor). Your analogy between University rankings and the Sumo banzuke is a very good one. I would say that the Univ. rankings are even more rigid than Sumo’s; I can’t imagine Harvard having to “retire”, because perceptions about these top-shelf institutions become self-generating, so Harvard will always be a Yokozuna because it’s always been a Yokozuna. Perhaps an even better analogy would be Sumo vs. University athletics programs; Oklahoma football or UCLA basketball dominated their “era” like the “Waka-Taka” or “Hakuho” periods in Sumo, but others (Alabama, Duke) eventually rise to replace them.
This rather runs contrary to your claim in another entry, that Canada has a remarkably equal university sector, with most people agreeing that you can get a good degree anywhere.
Maybe a better metaphor would be boxing, where the superheavyweights don’t compete against the bantam-weights, but also have less skill and nimbleness. But that would assume that speed, nimbleness and size are all worthwhile goals for universities.
Perhaps, but most of the rankings are measures of research, not quality of teaching. So the quality of teaching across Canada is relatively equal, while the research is wildly different depending on the purpose of the institution.
amazing written////keep it up….