You’ll recall that yesterday, in reference to the orgy of hype that accompanies the annual release of the THE World Reputation Rankings, I made the point that universities’ reputation really doesn’t change all that much on a year-by-year basis, and that, therefore, perhaps said orgy was a wee bit overdone.
When all was said and done, five universities (Arizona, Indiana, Leeds, U Zurich, and Tel Aviv) fell out of the rankings, with a similar number replacing them (Monash, Moscow State, Freie Universitat Berlin, New South Wales, and Maryland). Now, this might sound eminently stable – a 95% recidivism rate! But actually, having five positions in the top 100 shift in a single year is actually relatively volatile. What explains this?
Well, how about a change in the geographic balance of respondents? Unlike baseball’s all-star ballot, the THE rankings doesn’t discount voting for home-town favourites. And yes, stuff like that is happening. How else can you explain Turkey’s Middle East Technical University regularly cracking the THE’s top 100 when it doesn’t even make the top 500 in the ARWU?
THE isn’t consistent about the way it reports respondent populations, but here’s my attempt to compare the 2012 and 2013 samples:
Now, I’m sorry, but when 10% of respondents in a sample – which is meant to be globally representative – are from Australia and New Zealand (possibly more, depending on how that 12% “unspecified” pans out), you have a skewed sample. It beggars belief to know this and still claim with a straight face that Australia’s moving from 4 to 6 institutions in the top 100 actually means something. The least they could do is weight the results.
One final treat before I leave this topic for a few months. The keen-eyed among you may have noticed that the THE Reputation Ranking is based on two elements: a “research ranking” and a “teaching ranking”. The even keener-eyed among you may have read the fine print to learn that, in fact, the teaching reputation is a misnomer: it’s actually a ranking of “graduate teaching”, which you’d have to figure is so correlated to research as to makes no odds – and this is, in fact, the case. I plotted each institutions’ score for “Teaching” and “Research” Reputation on a scatterplot, and here’s what I found:
Yes, you’re reading that right. An R-square of .99. Kind of makes you wonder why they bother, doesn’t it?
Tweet this post
Thanks for this insightful analysis, Alex. Its very helpful for those of us who are called upon to explain “why we fell in the rankings…” to have some independent analysis.
Your “95% recidivism rate” seems like too low a bar to measure the stability of rankings. Of course a few always fall out of the top 100, but what would be really useful would be an aggregate measure of ranking instability, benchmarked against the ARWU’s measure of ranking instability. A simple rank correlation between the two years does the job, and provides a measure of how realistic the stability of various rankings really are.
Hi Walter. Glad you found this useful.
Good point. I’ll think about how to do that (not straightforward given the way everyone outside the top 50 is banded together).
A
Loved your scatter plot on “Teaching” and “Research” Reputation. They could have save a bundle…
As added information, you might be interested on my compilation of 9 world university ranking at http://listedtech.com/content/9-world-university-rankings-side-side-part-1
Have been wondering for a while why Australian/NZ are 10%. Proxy for Asia? Or because they have so many outstanding universities?
I’m Australian, and I could tell you that there’s six or seven Australian universities that could trouble the lower reaches of the AAU – if Melbourne or ANU were to rank 100 I would think it’s a little generous, 200 too harsh, but I couldn’t guess where they fall between. Six in the top hundred is garbage in, garbage out.
THE is a funny outfit – their readership being preponderantly UK, and it increasingly following Australian policy for the last twenty-five years, they may privilege what’s been done to de-academicise Australia’s never particularly academic approach to higher ed.
Obviously, they’ve essentially cut the US out of the loop (wouldn’t be surprised if Canada is overrepresented too, albeit not as badly as Australia).
And I guess the correspondence of research to teaching is meant to be some sort of triangulation device. Should keep that one under wraps…