HESA

Higher Education Strategy Associates

June 19

Europe’s Latent Strengths

I spent part of last week at the European University Association’s Funding Forum in Salzburg. Though it’s getting harder to see how you keep a European-wide conversation going when different countries are heading off in such different directions (small increases in funding in Germany and some Nordic countries, versus cuts of 35-45% in Ukraine and Greece), it was nevertheless a pleasant and productive event.

My job there was to give delegates a bit of a pep talk about European higher education, and why it may see better days soon. Sure, they have very big demographic and fiscal challenges, but these days, who doesn’t?

European universities have two big latent advantages over North American ones. The first is their cost structure. As we’ve seen before, European universities have done well to keep their labour costs relatively low. They also have room to squeeze a bit more productivity out of the teaching function by reducing the number of contact hours per degree. Though the numbers differ a bit from country to country, it seems that German and Austrian students, at least, have about 15% more contact hours on the way to a degree than do North American students. Close that gap, and that’s a lot of labour costs potentially saved.

Can this be done without reducing standards? Well, unlike universities here, European universities actually have some objective standards to uphold, thanks the widespread adoption of learning outcomes statements. As a result, I’d back their universities over ours every day of the week to engineer those kinds of efficiencies in a sensible way.

Public and Private Expenditures on Tertiary Education as a % of GDP, 2009

Then there’s the issue of income. European universities have an enormous untapped asset; namely, students. Even if EU members could close half the tuition revenue gap with non-EU OECD members, they would suddenly have enormous new pots of income which they can use to revitalize themselves. Almost instantly, they could go from having systems that are poor (if efficient), to having systems that are genuinely well-funded. The back half of this decade could be an exciting time in Europe, if governments and institutions have the will to grasp this nettle.

Of course, introducing tuition fees is a delicate thing, especially in countries where high unemployment is reducing the obvious payoff to higher education. Not surprisingly, I spent a lot of time there explaining what was going on in Quebec (most were shocked to find out how generous the Quebec government’s package really was). The lesson seems to be that introducing big changes in fee policy requires careful timing and – more importantly – governments with a lot of popular credibility. We might be waiting a while for that in Europe – and in Quebec, too, for that matter.

June 18

Uniquely Universal

Universities are astonishing, unbelievably resilient entities. Clark Kerr once noted that of the 75 Western institutions founded before 1520 (and which have survived intact to the present day), sixty of them are universities.

But universities aren’t merely unique in their reach across time – they are also unique in their reach across space. Few if any institutions are as truly global as a university. The basics of a campus are instantly recognizable whether you are in Nairobi, Tianjin or Regina. Give or take some nomenclature, administrative structures are essentially the same everywhere, and as David John Frank and Jay Gabler put it in their book, Reconstructing the University, they increasingly teach the same subjects and categorize knowledge the same way as well.

I was reminded of this the other day while reading James Fallows’ new book China Airborne, which examines both China’s enormous progress to date and its enormous challenges through the lens of the aviation industry. It’s an interesting book if you’re interested in innovation because it shows how tough it is to compete in so-called “apex” industries (that is, ones in which success requires the mastery of enormous numbers of different technological fields).

What caught my eye was Fallows’s discussion of how the Chinese reacted to having to adapt to new air safety regimes in the 1990s. They couldn’t be told they were adopting “American” standards, because that would have been humiliating. Being told they were adopting “international” standards was better, but what worked best of all was being told they were adopting international standards “with Chinese characteristics.” Being an ancient civilization (and they do genuinely think of themselves as a civilization rather than as a nation-state), it’s important for them to put their own imprimatur on things.

And yet, when it comes to universities, they don’t. China does have its own tradition of higher study dating back almost 1,400 years to the Great Academies of the Tang Dynasty which prepared students for imperial examinations; but while today’s gruelling Gaokao (i.e., university entrance) exams owe something to its imperial predecessor, there’s no pretence that universities are native Chinese or have Chinese characteristics. It’s all “global standards” and “world-classness” – without any modifications.

For all the criticisms and dissatisfaction which universities face in the West, it is in some ways the West’s most successful cultural export. Even the most virulent anti-colonialists never rejected them; indeed, they usually opened more. They have reached every corner of the globe and everywhere have a central place in the formation of the new middle classes. For all their faults, they have become the one universal and indispensable organization.

So if naysayers are getting you down, just remember – we must doing something right.

June 15

Bibliometrics Finale: Age and Size

Today, we use our H-index Benchmarking of Academic Research (HiBAR) to look at the relationship between institutional characteristics and H-index scores.

We’ve talked a lot this week about the positive correlation between a researcher’s age and his or her H-index score. But there’s another correlation to watch for: normalized institutional average H-index scores and institutional age. Check it out:

Normalized Institutional Average H-Index Score as a Function of Institutional Age

The result isn’t wholly clear cut: there are a lot of institutions that were created in the 50s and 60s which have surprisingly good normalized H-index scores, and a clutch of small liberal arts schools which have very low averages despite being quite old. However, overall, the relationship between age and normalized H-index score is negative.

This is interesting because it helps to demonstrate the degree to which institutional prestige – which is generally correlated with institutional age – has do with where talented academics try to locate. Academic salaries aren’t that different across the country; if academics only looked at money, one would expect a much smaller relationship between institutional age and average normalized H-index values. Basically, top researchers want to be where other top researchers are; and older, more prestigious institutions are always going to have a head start as far as concentrations of talent are concerned. It’s a virtuous circle – and one that’s very hard for new institutions to crack.

Another relationship we’re looking at is institutional size and average normalized H-index score. To wit:

Normalized Institutional Average H-Index Score as a Function of Institutional AgeSize

This is a much more straightforward result: big institutions have a lot more faculty members with long publication records. That may seem obvious, but it wouldn’t be the relationship that would hold in, say, the United States, where a lot of top institutions (e.g., Harvard, Yale) are quite small. In Canada, where the ability to pay for big-time research is dependent upon having a lot of undergraduates generating income that can be skimmed, it’s a much more direct relationship.

Stunningly, not a single university with less than 20,000 students has an average normalized H-index score above one (i.e., above the national average for all academics). This has some pretty significant implications for schools like Victoria and Saskatchewan, which have some significant research strengths but can’t generate sufficient revenue from undergraduate enrolment to make a really big push into the top league.

Now, the keenest-eyed among you may be looking at those two charts and wondering about those y-axis values. Are those really full-institution H-index values, normalized across all disciplines? Couldn’t somebody do a really interesting and unusually reliable research ranking with those?

The answers are yes and yes. But patience, grasshoppers: we’re not ready to roll it out just yet. You can spend the summer looking forward to it as a back-to-school treat.

June 14

Bibliometrics: Measuring Zero-Impact

Bibliometrics aren’t just useful for analyzing who’s being cited; they are also pretty good at telling you who’s not being cited, too.

Today, we’ll look at professors whose H-index (see here for a reminder of how it is calculated) is zero – that is, professors who have either never been published or (more likely) never been cited.

There are three reasons why a scholar might have an H-index of zero. The first is age; younger scholars are less likely to have published, and their publications have had less time in which to be cited. The second is prevailing disciplinary norms. there are some disciplines – English/Literature would be a good example – where scholarly communication simply doesn’t involve a lot of citations of other scholars’ work. The third is simply that a particular scholar might not be publishing anything of particular importance, or indeed publishing anything at all.

Let’s take each of these in turn. We can examine the first two questions pretty easily just by looking at the proportion of scholars with zero H-indexes by rank and field of study (our database has data on the rank of a little over three-quarters of academic staff in it – about 47,000 of the people in total, which is a pretty good sample).

Proportion of Academic Staff without a Cited Publication, by Rank and Field of Study

Surprised? So were we. Not because of the differences across ranks (H-index scores are necessarily positively correlated with length of career) or across fields of study (we did this one already0. What really blew us away was the number of full professors who have never had a paper cited, especially in the sciences. Who knew that was even possible?

So, what about that third reason? It is obviously difficult to generalize, but one should note that even within disciplines, there are some enormous gaps in publication/citation rates. In economics as a whole, 15.6% have an H-index of zero, but the proportion of economists in any individual economics department with an H-index of zero varies between 0% and 63%. In biology (disciplinary average: 7.7%), individual departments range between 0% and 60%; in history (disciplinary average: 13.4%), the range is between 5% and 50%. It is vanishingly unlikely that these differences are solely the result of different departmental age profiles; more likely, they reflect genuine differences of scholarly strength.

Now, there’s nothing saying all professors need to be publishing machines. But if that’s the case, maybe not all professors need to have 2/2 or 2/1 teaching loads to conduct all that impactful research, either. Running a university requires trade-offs between research and teaching: bibliometric analysis such as this is a way to make sure those trade-offs are well-informed.

June 13

Bibliometrics: Who’s the Best?

Today, we released the full version of our bibliometric paper, showing H-index averages on a discipline-by-discipline basis. You can find it here.

(Keep in mind while reading it that the H-index isn’t a wholly straightforward statistic to interpret. If one discipline has an H-index of 10 and another has an H-index of five, you can’t simply say that professors in one discipline publish twice as much as the other. An H-index is just the largest number of publications for which one also has at least the same number of citations – five papers with at least five citations gives an H-index of five, etc. So it’s possible that the discipline with an H-index of 5 might see just as many publications as the one with 10, just fewer citations.)

Now, because we know the H-index of every professor at the 71 institutions in our survey, we can aggregate their data on a departmental basis to find out how every academic department in the country fares against every department in the same field. Which means, basically, that we can pick out the “best” departments in each field, at least in the sense of the departments which have the highest average levels of scholarly productivity and impact, which is what the H-index measures. And we know how everyone loves rankings.

Do check part II of our paper for details, but the table below shows you which schools come out tops in each of the 55 disciplines we examined.

This is an interesting chart for a couple of reasons. First, the rank order of institutions with top departments doesn’t quite line up with the traditional perception of the rank order of top universities in Canada. In particular, UBC rarely gets top billing ahead of McGill and U of T, and Alberta isn’t usually seen as being behind Guelph and Saskatchewan. We wouldn’t read too much into that; even if it is the very best in just one field, Alberta is strong in a large number of fields (albeit many of them are in medicine, an area which our study wasn’t able to examine).

Queen’s standing in this listing is higher than it is in most research comparisons, but that’s because it does very well in disciplines which tend not to count much in traditional science-based indicator sets (e.g., literature and philosophy). When these fields are given equal weight, Queen’s does better.

A final point of interest is how widely excellence is spread: Lakehead and Wilfrid Laurier don’t usually make anyone’s list of major research universities, and yet, in certain areas of academia they can compete with the best.

More again tomorrow.

June 12

Bibliometrics: Canada’s Top Ten Science Faculties

We promise fun bibliometric data, we deliver fun bibiometric data. Today: we show you how to use H-index data to identify the top ten science faculties in Canada.

As we saw yesterday, science has the highest average H-index of any field; the average Canadian science professor has an H-index of 10.6. Recall that the H-index is the largest number of publications for which one also has at least the same number of citations – five papers with at least five citations gives an H-index of five, etc. But average H-indexes can vary enormously from one discipline to another even within a broad field of study like “Science.” Math/stats and environmental science both have average H-indexes below seven, while in astronomy and astrophysics, the average is over 20. Simply counting and averaging professorial H-indexes at each institution would unduly favour those institutions which have a concentration in disciplines with more active publication cultures.

(This, by the way, is true of pretty much all research metrics out there; even research dollars per professor can be skewed by these kinds of differences. Only the Leiden Rankings do any kind of field normalization, though they do not do use the H-index.)

The way to get rid of this bias is to replace raw H-index values with “standardized H-index scores.” These are derived simply by dividing one’s H-index score by the disciplinary average. Thus, someone with an H-index of 10 in a discipline where the average H-index is 10 would have a standardized score of 1.0, whereas someone with the same score in a discipline with an average of 5 would have a standard score of 2.0. This permits more effective comparisons across diverse groups of scholars because it allows them to be scored on a common scale while at the same time taking their respective disciplinary cultures into account. What this means in practice is that if one wanted to compare, say, science faculties, one can simply field-normalize all the scores, and then average them across all faculty members.

Our trusty database of over 60,000 faculty members allows us to easily do this for all of Canada’s science faculties. We find the top ten, in terms of concentration of scientific talent, to be as follows.

Just using a raw H-index, the University of Toronto comes first in part because it is better than UBC in some very high-citation disciplines like astrophysics. However, adjusting for disciplinary norms, UBC heads the list. Saskatchewan comes a surprising fifth in this list, thanks to strengths in environmental and earth sciences.

More tomorrow.

June 11

Back to Bibliometrics

About two months ago, we did a series on bibliometrics (if you missed it the first time out, you can catch up here), and promised we’d be back shortly with some new results. Well, we’ll those results will be released Wednesday, and we think they’re so interesting that we’ll be spending all week telling you about them.

For bibliometrics to be really useful, they need to (a) be able to capture information about both productivity and impact, (b) be easy to use and understand and (c) take account of differences in disciplinary publication cultures (or, to put it in the prevailing jargon, they need to be “field-normalized”). As we noted eight weeks ago, the H-index meets the first two criteria, but is still of only limited value because it doesn’t meet the third.

Until now. Over the past year, we at HESA have been building a database which contains the H-index data of over 60,000 professors employed at 71 Canadian universities in all fields of study except medicine (which we excluded on the eminently practical grounds that we can’t for the life of us distinguish teaching faculty from clinical faculty on institutional websites – if anyone wants to enlighten us, we can include them too). This is a pretty demanding job, as you can imagine: there’s a lot that goes into checking and double-checking publication records to make sure they all correspond correctly to individual professors. We then used these records to look at how H-index scores differ across fields of study.

When we release the full document Wednesday, you’ll be able to see the results down to the level of the individual discipline, but for now the main results are below:

Mean H-Index Scores By Field of Study

As you’d expect, professors in science, agriculture and engineering have significantly higher H-indexes than other disciplines. After that, you’ve got the social sciences, which come slightly ahead of the applied health disciplines (the publications profile of academics in dentistry, nursing, kinesiology, etc, is really nothing like those in the biological sciences), followed by business and the humanities. Below that, there is design & architecture and fine arts, neither of which really seems to use the printed word as the primary means of scholarly communications; the median professor in these disciplines has either never published or never been cited in a scholarly publication.

You might think of this as a bit of a “so-what” finding. But it’s a big deal for two reasons: first, no one has ever done this systematically with the H-index before and second, now that it’s been done, it opens up an enormous amount of potential research on research and impact to be conducted – and we’ll be giving you a taste for the rest of the week.

– Alex Usher and Paul Jarvey

June 08

Getting a Global Common Data Set Off the Ground

How could a global common data set (CDS) come into existence? Here are a few considerations:

In addition to improving accuracy and comparability, common data sets come into existence for two reasons. The first is to save money by limiting the number of data requests flying in from every yahoo wanting to create his or her own ranking. The second, less obvious reason, is that the creation of an open-access data platform lowers the barriers to entry for new rankers. It’s a way of busting data monopolies.

For that reason, it’s unlikely that a commercial entity is going to lead the way, except as a contract provider of technical assistance. The real impetus is going to have to come from institutions themselves. But which institutions?

An initiative backed by schools from a single country or region probably wouldn’t get very far, which is why I’m not sure U-Multirank is the way forward. But formal global bodies aren’t the answer either. IAU and UNESCO have too many members wishing to play King Canute to the rising tide of rankings for them to be an option.

Ad-hoc coalitions of institutions are possible: if you were to throw together some interested IR people from (say) Australia, Canada, Singapore and Norway, you could see it happening. But structure matters in collaborations, which is why a meeting of minds at two or three of the big existing alliances like the International Alliance of Research Universities or Universitas 21 would seem like the likeliest source of a successful project. In terms of optics, locating project management outside the U.S. and U.K. would be preferable.

The keys to success will be flexibility and cost-effectiveness. If a Global CDS asks for too much data from its members, or asks for data which is too difficult to obtain or manipulate, it will never catch on. Things like research output, student survey results and more exotic ideas like “artistic output” need to be sidelined for the moment (or at least relegated to optional data fields outside the “core”).

The important thing will be the core variables: money, students, faculty. Those are things anyone can count, but which for various reasons are compiled differently around the globe. If these elements can be well-defined and well-managed, the resulting success will bring in new members and, over time, an appetite to increase the project’s scope.

Basically, creating a genuinely globally CDS is a ton of work. Thomson-Reuters or U-Multirank would have to get someone really ticked off for people to think it worth the investment. But the proliferation of university alliances are reducing the transaction cost of this kind of collaborative work every day. This could come sooner than you think.

June 07

What Would a Global Common Data Set Look Like?

The discussion Kris Olds and I started a couple of days ago (see here and here) about a global common data set seems to have generated quite a bit of discussion, so I thought I would flesh out two sets of thoughts regarding what a Global CDS would need to look like – today: content; tomorrow: governance.

Start with first principles on content: it needs to be common enough that most institutions around the world are able to produce it. That necessarily means that, to start at least, the data will be skewed towards things that are relatively easy to count, not necessarily things that matter.

What would qualify? Student data, for one thing – graduate and international student numbers are always interesting, but in terms of student-teacher ratios, not just numbers but total course hours should be calculated. Academic staff, for another – though, as per yesterday’s discussion, one would need to set out data for a number of different types of faculty (full-time, adjunct, etc.) to allow for different types of comparisons. Strictly defining these different categories and ensuring all institutions can distinguish properly between them will be paramount. And, of course, as with students the faculty data shouldn’t just be raw numbers – course hours, too, are important.

Financial data will be important, as well – income from various sources (especially research income) will need to be included, as will some expenditures like libraries. But because universities in different parts of the world have radically different abilities to control their own budgets this data will need to be complemented with information about various dimensions of university autonomy (the recent European Universities Association survey on autonomy is a good model).

Beyond that, it might get pretty thin. There just isn’t enough university-produced data in common across international boundaries to go much further. Within Europe, for instance, there might be some desire to bring indicators such as “regional joint research projects” which seemed to work in the U-Multirank project; similarly, North American schools might want NSSE or CLA data. Leaving room for optional indicators might be a smart way to go.

More speculatively, there might be ways to combine data from national student satisfaction surveys in places like Canada, Germany and Taiwan into a single global, super student-indicator; and there might also be interest in including various forms of bibliometric information (though this can easily be done outside a CDS as well) – but now we’re talking about things a long way down the road.

So how might someone create and manage such a possibly unwieldy global database project? Tune in tomorrow.

June 06

Counting Faculty, Counting Students

Though amateur higher education statisticians are addicted to it, there is virtually no statistic less useful than the student-staff ratio. There are basically two reasons why this is the case.

The first is that not all students are alike. Some are full-time, some are part-time. This problem is reasonably easy to solve by creating a method for calculating full-time equivalency. But for this, the number of credit-hours students take must be transparent. The second is that not all professors are alike. This is a bigger issue, because it takes in two possible issues.

The first is that the definition of a “professor” is more fluid than most people realize. Everyone can agree to including full-time tenured or tenure-track professors. But what about adjuncts? Chargés de cours? Clinicians in teaching hospitals? Emeritus professors? When outside agencies come knocking for statistics from universities, universities will supply figures that depend on their use. If they want to emphasize how underfunded they are, they will use the smallest possible number; if they want to emphasize how great the student-professor ratio is, they will include all of the above. The number of professors on hand can shift considerably in the context of a single data request.

There’s a solution to this of course – schools can simply report many different categories of professors and allow users to use whichever combination of them they choose. Ontario universities, for instance, report about a half-dozen such figures to government for precisely this reason. But that only solves part of the problem. Because, just as not all students are in class for the same amount of time, not all profs spend the same amount of time in class, either.

The reason anybody cares about staff-student ratio is that it’s a proxy for class size. But if you have two universities with identical ratios, but at university A the expected course load is 4/4 and at university B its 2/2, you’re going to have radically different class sizes. Just counting bodies isn’t enough: you need to count hours in the class.

Here’s what institutions ought to do. They ought to publish the number of credit hours taken by students. And they ought to publish the number of class hours that each of the various types of professors (tenure-track, adjunct, etc.) has. If they could do this at the faculty level, so much the better. Then, by dividing the two sets of numbers, you’d get an actual average class size. Which, again, is what people actually care about.

Making this data available would vastly improve our understanding of classroom conditions. And at most institutions, it should not require any extra data collection – everything required should already be “in the system.”

So, to all our IR readers: how about it?

Page 20 of 40« First...10...1819202122...3040...Last »