Category Archives: research

May 22

Bad Arguments for Basic Research

Last week’s announcement that the NRC was “open for business” has, if nothing else, revealed how shockingly weak most of the arguments are in favour of “basic” research.

Opponents of the NRC move have basically taken one of two rhetorical tacks. The first is to present the switch in NRC mandate as the equivalent of the government abandoning basic science. This is a bit off, frankly, considering that the government spends billions of dollars on SSHRC, NSERC, CIHR, etc. Even if you’re passionate about basic research, there are still valid questions to be answered about why we should be paying billions of dollars a year to government departments doing basic research when the granting councils fund universities to ostensibly do the same thing.

The second argument is to say that government shouldn’t support applied science, because: a) it’s corporate welfare, and b) all breakthroughs ultimately rely on basic science, and so we should fund that exclusively. It seems as though those who take this line have never heard of Germany’s Fraunhofer Institute, a publicly funded agency in Germany which does nothing but conduct applied research of direct utility to private enterprises. It’s generally seen as a successful and useful complement to the government’s investments in basic science through the Max Planck Institute, and to my knowledge, Germany has never been accused of being anti-science for creating and funding Fraunhofer.

Another point here: the benefits of “basic” research leak across national borders. Very little of the upstream basic research that drives our economy is Canadian in origin. So while it’s vitally important that someone, somewhere, puts a lot of money down on risky, non-applied research, individual countries can – and probably should – make some different decisions on basic vs. applied research based on local conditions.

The relative benefit of a marginal dollar investment in applied research vs. basic research depends on the kind of economy a country has, the pattern of firm size, and receptor capacity for research. It’s not an easy thing to measure accurately – and I’m not suggesting that the current government has based its decision on anything so empirical – but it’s simply not intellectually honest to claim that one is always a better investment than the other.

Opposition to the NRC change is clearly – and probably justifiably – coloured by a more general irritation at a host of this government’s other policies on science and knowledge (Experimental Lakes, long-form census, etc). But that’s still no excuse for this farrago of flimsy argumentation. Rational policy-making requires us to engage in something more than juvenile, binary discussions about what kind of research is “best”.

May 08

Fundamental Research

“Scientific discovery is not valuable unless it has commercial value” (John McDougall, NRC president, yesterday).

“Discovery comes from what scientists think is important, not what industry thinks is important. Fundamental scientific advancement drives innovation, and that is driven by basic research.” (David Robinson, CAUT Associate Executive Director, yesterday).

Some days, the level of discourse in Canadian higher education policy seems to be improving. Other days, like yesterday, it is full of childish, one-dimensional arguments about the nature of science and research, arguments that the rest of the world outgrew of fifteen or twenty years ago, and I just want to weep.

The basic concept of research was invented by Vannevar Bush in his 1945 work, Science: The Endless Frontier. In order to press for greater funding of university research, Bush made a sharp distinction between “basic” (or “fundamental”) research, “performed without thought of practical ends” at universities, and “applied” research” (something to be left to business and the military) that developed from the former. To have more of the latter, he conveniently argued, you needed more of the former.

But this neat division was a rhetorical device rather than a meaningful scientific taxonomy. As Donald Stokes pointed out in his book, Pasteur’s Quadrant, outside of theoretical physics, there really aren’t many fields of science where scientists knock about “without thought of practical ends”. Fundamental research often solves very practical problems that industry faces (which is true for a great deal of research in Engineering, Computer Science, and Chemistry), or which quite clearly has commercial applications (true for much medical research, for instance). Discovery, as David Robinson says, does come from “what scientists think is important”, but that begs the question: “how do they decide what’s important”? The answer, often, is discovered by interacting with industry and finding out what companies think is important. If that weren’t true, frankly, the contribution of university science to economic growth would be a hell of a lot smaller than it is.

As for the notion that scientific discovery is not valuable without a commercial application: man, that’s some strong ganja they’re smoking on Montreal Road. Are mathematics worthless because you can’t patent an equation? Was Galileo just some flâneur because he never made a penny off heliocentrism? How the hell can you tell, a priori, whether something has a commercial application? I mean, Rutherford wasn’t thinking about multi-billion dollar industries in telecommunications, nuclear power, and quantum computing when he did his gold foil experiments. Yet all those industries would be non-existent if we still thought that atoms were solid shells.

As a country, our scientific and academic leaders should do better than this.

April 25

The Leiden Rankings 2013

Though it was passed over in silence here in Canada, the new Leiden university research rankings made a bit of a splash elsewhere, last week. I gave a brief overview of the Leiden rankings last year. Based on five years’ worth of Web of Science publication and citation data (2008-2012), it is by some distance the best way to compare institutions’ current research output and performance. The Leiden rankings have always allowed comparisons along a number of dimensions of impact and collaboration; what’s new – and fabulous – this year is that the results can be disaggregated into five broad areas of study (biomedical sciences, life & earth sciences, math & computer science, natural sciences & engineering, and social sciences & humanities).

So how did Canadian universities do?

The big news is that the University of Toronto is #2 in the world (Harvard = #1) in terms of publications, thanks mainly to its gargantuan output in biomedical sciences. But when one starts looking at impact, the story is not quite as good. American universities come way out in front on impact in all five areas of study – natural, since they control the journals and they read and cite each others’ work more often than they do that of foreigners. The UK is second in all categories (except math & computer science), third place in most fields belongs to the Dutch (seriously – their numbers are stunning), followed by the Germans and Chinese, followed (at a distance) by Canada and Australia. Overall, if you look at each country’s half-dozen or so best universities, sixth or seventh is probably where we rank as a country in all sub-fields, and overall.

Also of interest is the data on collaboration, and specifically the percentage of publications which have an international co-author. That Canada ranks low on this measure shouldn’t be a surprise: Europeans tend to dominate this measure because there are so many countries cheek by jowl. But the more interesting finding is just how messy international collaboration is as a measure of anything. Sure, there are some good schools with high levels of international collaboration (e.g. Caltech). But any indicator where the top schools are St. Petersburg State and King Saud University probably isn’t a clear-cut measure of quality.

Among Canadian schools, there aren’t many big surprises. Toronto, UBC, and McGill are the big three; Alberta does well in terms of volume of publications, but badly in terms of impact; and Victoria and Simon Fraser lead the way on international collaborations.

If you have even the slightest interest in bibliometrics, do go and play around with the customizable data on the Leiden site. It’s fun, and you’ll probably learn something.

April 05

No to “World-Class” Research in the Humanities

You often hear talk about how Canadian institutions need to do more research. Better research. “World-class” research, even. Research that will prove how smart our professors are, how efficient they are with public resources, and, hence, justify a claim to an even greater share of those resources.

In medicine, the biological sciences, and engineering, this call is easy to understand. Developments in these areas can – with the right environment for commercialization – lead to new products, which, in turn, have direct economic benefits to Canadians. In the social sciences, too, it makes sense. Most social sciences have (or should have) some relevance to public policy; thus, having world-class research in the social sciences can (or should) mean an improvement in that country’s governance, and its ability to promote a strong, healthy, and equitable society.

But what about in the humanities? Is there a national public interest in promoting world-class research in the humanities?

My answer is no. For two reasons.

The first is kind of technical. When it comes to research, “world-class” status tends to get defined by bibliometrics. In the sciences, scholarly conversations are, by their nature, global, and so a single standard of measurement makes sense. But in the humanities, an awful lot of the conversations are, quite properly, local. And so while bibliometric comparisons in the humanities, within a single country (say, between institutions), might say something important about relative scholarly productivity, comparisons between countries are, to a large degree, only measuring the relative importance of different national polities. A strategy favouring world-class bibliometric scores in History, for instance, would de-emphasize Canadian History and Aboriginal studies, and instead focus on the Roman and British Empires, and the United States. And that, obviously, would be nuts.

But there’s a bigger issue here: namely, why do we assume that the worth of humanities has to be judged via research, in the same manner we judge scientific disciplines? Arguments in defence of the humanities – from people like Martha Nussbaum, Stanley Fish, etc. – stress that the discipline’s value is in encouraging students to think critically, to appreciate differences, and to create meaning. And it’s not immediately obvious how research contributes to that. Even if you completely buy the argument that, “scholarly engagement is necessary to teaching”, can you really claim that an increased research load improves teaching? Have students started thinking more critically since 3/3 teaching loads were cut to 2/2 in order to accommodate more research?

The real national public interest is in having a humanities faculty that can develop critical thinkers, promote understanding, and foster creativity. Figuring out how to better support and celebrate those things is a lot more important than finding yet more ways for the humanities to ape the sciences.

January 17

Can’t Get No Satisfaction (Data)

Many of you will have heard by now that the Globe and Mail has decided not to continue its annual student survey, which we at HESA ran for the last three years. The newspaper will continue publishing the annual Canadian University Report, but will now do so without any quantitative ratings.

Some institutions will probably greet this news with a yawn, but for a number of others, the development represents a real blow. There were a number of institutions who based a large part of their marketing campaigns around the satisfaction data, and the loss of this data source makes it more difficult for them to differentiate themselves.

When the survey started a decade ago, many were skeptical about the relevance of satisfaction data. But slowly, as year followed year, and as schools more or less kept the same scores in each category year after year, people began to realize that satisfaction data was pretty reliable, and might even be indicative of something more interesting. And as it became apparent that satisfaction scores actually had a reasonably good correlation with things like “student engagement” (basically: a disengaged student is an unhappy student), it also became apparent that “satisfaction” was an indicator which was both simple and meaningful.

Sure, it wasn’t a perfect measure. In particular, institutional size clearly had a negative correlation with satisfaction. And there were certainly some extra-educational factors which tended to affect scores, be it students’ own personalities, or even just geography – Toronto students, as we know, are just friggin’ miserable, no matter where they’re enrolled. But, when read within its proper context (mainly, by restricting comparisons to similarly-sized institutions), it was helpful.

Still, what made the data valid and useful to institutions was precisely what eventually killed it as a publishable product. The year-to-year reliability assured institutions that something real was being measured, but it also meant that new results rarely generated any surprises. Good headlines are hard to come by when the data doesn’t change much, and that poses a problem for a daily newspaper. The Globe stuck with the experiment for a decade, and good on them for doing so; but in the end, the lack of novelty made continuation a tough sell.

So is this the end of satisfaction ratings? A number of institutions who use the data have contacted us to say that they’d like the survey to continue. Over the next week or so, we’ll be in intensive talks with institutions to see if this is possible. Stay tuned – or, if you’d like to drop us a note with your views, you can do so at, info@higheredstrategy.com.

October 26

Research Rankings Reloaded

You’ll recall that a couple of months ago we released a set of research rankings; you may also remember that complaints were raised about a couple of issues in our methodology. Specifically, critics argued was that by including all permanent faculty we had drawn the net too wide, and that we should have excluded part-timers.

Well, we’ve now re-done the analysis, and are releasing them today as an annex to our original publication for all to see. Two key things to highlight about the changes are (i) the effect of excluding part-time professors is more significant in SSHRC disciplines than in NSERC-ones, and (ii) the use and function of part-time professors appear seems to differ systematically on linguistic grounds. Compared to part-time professors at anglophone universities, those at francophone universities are both more numerous and have profiles which resemble those of adjuncts at anglophone institutions (whereas in Anglophone institutions, part-timers look reasonably similar to the full-time population).

At the top of the table, not a great deal changes – it’s still the same top six in both SSHRC and NSERC totals (though ordinal position does change slightly – McGill slips ahead of UBC to top spot in the SSHRC rankings because of much better performance on research income). Neither is there much change in the bottom quartile or so. In the middle ranks though, where institutions are more tightly bunched together, small changes in scores can cause some pretty big changes in rankings; University of Manitoba goes from 27th to 17th in the NSERC rankings; UQAM (which has by far the country’s largest contingent of part-time faculty) jumps from 43rd to 17th in the SSHRC ones. In fact, francophone institutions generally did a lot better with the revisions. What we had initially assumed was a “language effect” – poor results driven by the fact that publishing in French limits readership and hence citations – may in fact have been driven by employment patterns.

But for most institutions, contrary to the expectations of some of our critics, not much changed. Even where it did, gains in the one field were sometimes offset by losses in the other (Ottawa, for instance, rose five places in Arts but fell two in Science). Which makes sense: the only way excluding part-timers would change outcomes would be if your institution used part-timers in a significantly different way than others do.

Now, you may still dislike our rankings because you don’t like field-normalization, or the specific metrics used, or whatever. That’s cool: debating metrics is important and we’d love to engage constructively on that. But let’s bury the canard that somehow an improper frame significantly skewed our results. It didn’t. The results are the results. Deal with ‘em.

October 16

Is Research Getting Harder

I was reading through Paula Stephan’s How Economics Shapes Science - which is, by the way, an utterly fantastic book for anyone who wants to understand how universities actually work – when I came across this interesting little table.

Average Number of Co-authors per Paper

Over the space of nearly 20 years, the average number of co-authors per article has increased across all fields of study, albeit not uniformly (the effect seems bigger in the physical sciences). But why is this happening exactly?

Improvements in IT are one possible answer. A cynic might point to the increased importance of publication in the tenure process – more co-authors per article means less work per author, which in turn makes it possible for each individual scholar to produce more publications. A third answer is less cynical but perhaps more troubling: it may simply be that the overall productivity of academic researchers is declining.

We know that over the decades, the degree of researchers’ specialization has increased and the time to Ph.D. graduation has crept up. There’s simply an increasing amount of knowledge to be absorbed before one can be considered to be at the cutting edge. Newton famously said he stood on the shoulders of giants; these days, doctoral students have to clamber up a large pyramid of giants in order to get to the point where they can become fully-fledged members of the academy.

The trend to specialization is a way of dealing with this problem – it cuts the width (if not the depth) of what doctoral students need to know in order to be given their diploma. But since worthwhile – i.e., publishable – research usually crosses these sub-sub-sub-disciplinary boundaries, the trend to specialization has led to a decline in the ability of individual researchers to complete publishable research independently, and in turn requires increasingly large teams in order to obtain publishable results.

We can’t be completely sure about what’s going on because to my knowledge the necessary bibliometric studies haven’t been carried out. If the tendency for larger authorial teams is actually being offset by increased publications per professors, then there’s nothing to worry about because it implies that specialization is doing what Adam Smith and his pin factory said it would (that cynical explanation I provided earlier? It’s actually the hopeful scenario).

If, on the other hand, the number of publications per professor is not increasing as fast as the number of authors per paper, then we may really be hitting some kind of productivity wall. And that’s probably not good over the long-term for the logic underlying both the modern research university and graduate education as a whole.

September 26

Public Research or Risky Research?

Why do we pay for research from the public purse, exactly? As I wrote a few weeks ago, it wasn’t always the case. It was only after American scientists working in universities demonstrated how their knowledge and skills could contribute to national security that the idea really took off.

Fifteen years later, two American economists came to provide a dollars-and cents rationale for public funding of research. In 1959, Richard Nelson argued that the private sector was likely to underinvest in “basic research” with wide applications relative to “development” of specific products and applications because companies were simply much less likely to be able to fully capture the benefits of the former. Kenneth Arrow then popped up a couple of years later in a paper called The Rate and Direction of Inventive Activity and argued that there were actually three reasons why companies might not invest appropriately in research: indivisibility, inappropriability and uncertainty.

When we hear university lobbyists talk about the need for more “investment” in research, they’re usually relying on Nelson’s arguments, which imply that basic research is a classic case of government intervention to solve a market failure. But maybe we should pay more attention to Arrow’s arguments; the private sector doesn’t shun basic research because it’s unprofitable – it shuns it because it’s risky.

Yet, as governments around the world have increased their investments in research, they have also been promoting ideas about ensuring that the public receives “value-for-money.” The problem is that doing this creates a lot of incentives for “safe” research – research that one knows in advance has a good chance of “success”, in the sense that it will yield a modest advance in human understanding (and, of course, publishable results).

The problem is that the more governments insist on “value for money,” the less useful public funding actually is. If government-funded science is just as risk-averse as private science, what’s the point? Obviously all that research money is useful for the care and feeding of twenty thousand scientists or so, but what’s the actual public benefit?

There’s an interesting thought experiment here: what if we slashed research budgets but also removed all the constraints around value-for-money? There’d be less money overall, but just the crazy geniuses would be let loose to do things which are really innovative. Sure, there’ll be higher rates of failure but that’s what public funding is actually for.

What do you think the actual benefits to society would be? Would they increase or decrease? Hw big a research budget cut would it take before the loss of money outweighed the gain of losing the constraints. In other words, what’s the cost of “value-for-money?”

September 19

Prizes for Excellence

I wrote recently about using prizes as a way to distribute research money. More generally, though, prizes have a lot of potential as a way for governments to influence institutional behavior and create a more diverse higher education sector, and deserve to be given a lot more thought by policymakers.

The reason for this is that we desperately need a more diverse set of incentives in our system. When politicians moan about how universities “aren’t responsive,” they are getting it precisely backwards; universities are plenty responsive to the financial incentives that are in front of them. At the moment, they only really get rewarded for doing two things: admitting more students and publishing more research (academic norms of behavior reward publications, too, so there’s a built-in second incentive on that front). Ministers can kvetch all they want about other stuff, but unless you change the incentives facing institutions, their behavior won’t change.

The solution, simply, is to set up a different set of incentives. They don’t have to be enormous – a small tail can wag a pretty big dog in academia – but they have to substantial and varied. For instance, why not give $5 million each year to the institution that gets the best teacher ratings from its students (yes, I know they can be gamed, but there are equally ways to control for gaming – see Côté and Allahar on this.). Or $10 million to the institution doing the most to promote local economic development? Or $10 million to the university doing the best job of integrating technology in the classroom? It’s a cheap way to buy change.

A really ambitious version of this was mooted recently in the pages of Policy Options, where former PMO Chief of Staff Ian Brodie suggested creating a $1 billion challenge fund to get one Canadian institution into the Top 10 of one of the major rankings agencies. Under his plan, the institution would get the money up front and pay it back if they didn’t succeed within 10 years.

Now, I’m fairly sure this specific plan wouldn’t work. Apart from the Shanghai rankings, the methodologies change too radically from year to year to make them reliable barometers of excellence, and the gap between U of T, UBC and McGill and the current top 10 of the Shanghai rankings is probably too big to be bridged by $1 billion over ten years. And if I were going to start incentivizing stuff, it probably wouldn’t be in research, because there are already lots of incentives there.

But Brodie’s basic idea is an excellent one. We need more and varied incentives. Bring’em on.

September 06

A Response to Critics

So, we’ve been hearing a number of criticisms – both directly and via the grapevine – of the research rankings we released last week. (Warning: if you’re not entranced by bibliometric methodology, you can safely skip today’s post).

The main point at issue is that at some schools, our staff counts appear to be on the high side. Based on this, some schools have inferred that we are judging them too harshly – that if we had fewer observations, the denominator would be smaller and their score would rise. But this is not quite correct.

There are two possible reasons why our staff counts are high. The first is that we do double-count people who are cross-posted across departments. But that’s a feature, not a bug. We aren’t taking one h-index score and dividing it across two observations – that would be silly. Instead, we calculate someone’s normalized h-index twice and then average them.

Say Professor Smith, with an H-index of 4, is cross-posted between discipline A (where the average H-index is 5) and discipline B (avg. H-index = 3). This person’s normalized score would be .8 (4/5) in discipline A and 1.33 (4/3) in discipline B. When aggregating to the institutional level, both scores are included but due to averaging this person would “net out” at (.8+1.33)/2 = 1.065. If they were only counted once, they would need to be assigned a score of either .8 or a 1.33, which doesn’t seem very sensible. Thus, to the extent that high staff numbers are due to such double-counts, we’re confident our methodology is fair and doesn’t penalize anyone.

The second possibility is that errors were made in harvesting faculty names from 3500-odd departmental websites. Some mistakes are probably ours, but a major factor seems to be a widespread practice of schools not distinguishing between permanent and part-time faculty on their websites. To the extent that the misidentified staff are graduate students or post-docs, then miscounts will lower institutional scores. However, to the extent that the misidentified staff are adjuncts – especially ones who are recently retired faculty or distinguished practitioners from outside the academy – then our miscounts may actually be inflating institutional scores. Smaller denominators don’t necessarily mean higher scores.

With the help of one affected institution, we’re trying to work out what the issue is and whether the problem in fact affects scores significantly. Since we believe in the importance of transparency, accuracy and accountability (see the Berlin declaration on rankings, which I helped draft), we’ll extend the offer to all institutions who feel our methodology has portrayed them inaccurately. If we find a problem, we’ll correct it and publish results here.

You can’t be fairer than that.

Page 1 of 312 3