You may have seen stories in Inside Higher Ed and University World News about the QS World Rankings, and specifically, a claim by a Senior Researcher at Berkeley named Igor Chirikov that QS’s conflicts of interest “may produce significant distortions in global university rankings”.  Cue much clucking on the interwebs about issues with rankings.

All I can say, having read the paper, and having some idea of what QS does, is that the word “may” is doing a fair bit of work in that sentence.  And I am a little bit disappointed in the credulity with which IHE and UWN have shown in reporting this paper’s conclusions straight-up.

Here’s my read of what the paper asserts:

1. QS ranks universities. It also sells a variety of services to universities, including advising on how to improve in rankings and QS Stars. This presents a potential conflict of interest.
2. Using publicly-available data on Russian universities, one can distinguish between nine institutions which buy a lot of services from QS (hereafter, QS-heavy) from those which do not (QS-light).
3. Upon examination of these data, one sees that QS-heavy institutions seem to have improved more in the rankings over the period 2016-2021 than QS-light ones. Specifically, QS-heavies seem to have significantly improved their reported student-faculty ratios compared to the QS-lights.
4. Data on the same institutions from the Times Higher Education rankings, with respect both to change in rankings and change in student-staff ratios, do not show the same kind of division between QS-heavy and QS-light institutions. Neither, with respect to student-staff ratios at least, do data from the Russian Ministry of Higher Education.

From this, the author contends that “universities that use services provided by ranking companies more frequently may improve their positions in  global  university  rankings  over  time,  regardless  of  improvements  in  their  institutional quality.”

There are four issues here.  First, Chirikov chooses not to name which 9 institutions that meets his definition of QS-heavy.  That makes it unnecessarily difficult (given that this data is apparently all in the public realm in Russia) for anyone to verify his findings.  Second, there is no attempt to break down what specific QS-services were purchased.  One would think that one might try to make some distinctions between purchases of booth space at QS Events, or advertising on the QS website, or actual consulting services.  But no, a dollar is a dollar: any money to QS for any purpose is equally suspect.  Third, there is no attempt to look at the timing of payments and link them to changes in rankings.  This leaves the possibility of reverse causality – that perhaps some of these institutions spent more on advertising in the latter half of the quinquennium after improving in the rankings in the early years.

Fourth – and I think this is a big one – comparing changes in ranks between THE and QS is infeasible because THE introduced a methodological change during the period under study that had serious negative repercussions on a lot of Russian universities.  specifically, it introduced “fractional counting”, meaning that an article with one author counts as a score of one, but one with a hundred authors from a hundred different institutions counts as a score of 1/100.  This affected several Russian universities that with many physical sciences faculties where co-publication with dozens or hundreds of co-authors is the norm.  Specifically, Tomsky Polytechnic, Kazan Federal University, Novosibirsk State University and Saint Petersburg State University all got creamed, falling between 250 and 650 spots in the rankings.  QS made no such methodological change.  As a result, the comparison being made here is highly suspect. 

I don’t have access to the data Chirikov does.  My access to QS data is limited back to 2018, and there is no public access to raw student-staff numbers from QS (though scores are available) or to student-staff scores from THE (though raw data is available).  That said, I can see at least some of what he is analyzing. There are four institutions which a) are ranked in the top 1000 institutions worldwide in both QS and THE, b) saw QS rankings increased significantly (90+ places) between 2018 and 2021 and c) reported student-staff ratios that seemed to head in opposite directions for QS and THE.  Those institutions are: Ural Federal University, National University of Science and Technology, the People’s Friendship University of Russia, and ITMO University in St. Petersburg, plus a fifth: Far Eastern University in Vladivostok, which meets the second and third criteria. 

If these institutions are among Chirikov’s QS-heavies, it might be significant.  But three other institutions – Tomsk University, the Moscow Institute of Physics and Technology, and the Moscow Engineering and Physics Institute (also sometimes called the National Research Nuclear University) – all showed a similar dissonance in the way student-staff ratios were reported to the two agencies, but unlike the other institutions, saw no improvement in their overall rank.  This suggests that the link between student-staff ratios and rank may not be as regular as Chirikov implies.

For what it’s worth, I’ve been at several events in Russia, organized by the Polish Perspektywy Foundation on behalf of the organization in charge of the Russian university excellence program at which QS representatives were also present.  I’ve therefore a reasonably good idea of what happens in these “consulting” sessions: they are mostly about explaining how individual rankings and their indicators work.  Obviously, if someone is explaining indicators and weights to you, it doesn’t take a lot of imagination to pick specific indicators to improve upon.  And to say that rankings systems have difficulty imposing uniform definitions of either student counts or staff counts on institutions is a massive understatement.  So, it’s not out of the question that those universities who took the time to pay QS (as well as perhaps others) to better understand how rankings are calculated, and where there is room for – shall we say – some creativity in data submission – acted upon that knowledge.  And if you wanted to, you could call that a form of corruption. 

But we have no idea if the institutions that saw a big change in student-staff ratios reporting were the “QS-heavy” ones.  We have no idea what services the QS-heavies purchased.  The link between the change in the student-staff ratio and the change in the overall rank doesn’t seem to be especially tight.  And using the Times Higher Education Supplement as a control group, which in theory is a very good idea, is undermined by a shift in the THE’s methodology which makes it highly non-comparable.

What you’re left with, really, is a claim that it is unethical for an organization which ranks universities to also accept money from them, as it necessarily places them in a conflict of interest.  And you know, fair enough.  But this would be true regardless of whether one could “prove” an effect: conflict of interest is all about perceptions. It exists whether or not someone committed an abuse of power by intervening to alter a result.  People sometimes confuse the two, but they are fundamentally different.

So yes, it’s fine to claim QS is in a conflict on rankings.  But claiming that this results in abuse, and that this is demonstrable from the experience of a couple of dozen (unidentified) Russian universities?  This paper is far from a smoking gun.