Yesterday, we looked at how Economic Impact Statements are put together.  Today, we want to look at the uses and misuses of these statements.

Let’s start by acknowledging that these statements are not primarily designed to be objective, academic analyses of impact.  Rather, they are political documents, meant to put an institution in a good light.  There’s nothing wrong with that, but it means that they need to be read with a certain eye.  Given the built-in incentive to exaggerate the figures, it’s important to know which exaggerations are honest and which ones are chicanerous.

The first honest exaggeration involves the economic footprint of the institution: specifically, the idea that “benefits” are somehow equivalent to costs times some multiplier (we’ll get back to the multipliers in a second) which can then also be converted into a number of jobs created.  It’s not that this number is wrong, exactly: it’s a pretty standard Keynesian way of describing the impact of an institution on a local economy.  The issue is that pretty much every university has more or less the same effect.  Provided you are using the same multiplier (which, really, you should be), saying that University X has a $1 billion impact on the economy and University Y has a $500 million dollar impact does not mean one is twice as beneficial as the other, it just means one has twice the costs of the other.   And, of course, it says nothing about opportunity costs (i.e., what benefits similar investments in other areas might bring).  But since everyone does exactly the same thing (not just in education, but in other fields as well), this is what I call one of the honest exaggerations.

It’s the same thing, broadly speaking, with trying to measure graduate earning premiums.  There is certainly room for both chicanery and error in terms of how you come up with a number which represents average graduate earnings.  But calculating the premium – that is, the proportion of those earnings that you can then claim as your institution’s “value added” tends to be handled in a similar manner everywhere: just subtract the average amount earned by high school graduates.  This is preposterous, because high school graduates aren’t exactly an equivalent control group (selection effects matter).  But again, no one really knows how to calculate a better counterfactual and everyone uses the same practice, so it doesn’t give anyone an advantage, per se.

The problem, rather, is when institutions engage in what one might call “competitive counting” or “competitive multiplication”.  Here are a few examples.

1. Using bigger multipliers. As noted yesterday, there is some legitimate debate about what the “correct” multiplier for institutional spending is. But at most Canadians institutions, it is a number adjacent to 1.5 and indeed a number of institutions simply use the figure 1.5 to simplify matters. But occasionally, someone substitutes a different number. Trent University, for instance, uses 2.0 as the multiplier. No rationale given. It certainly makes the institution look better, but it’s not actually a reflection of better performance. This is maybe the purest example of “competitive multiplication”.
2. Exaggerating student expenditures (1). Dollars spent by International and out-of-province are certainly attributable to the institutions, because without the institution, those dollars would likely not be spent in-province. But what about local or in-province students, who for the most part would presumably be spending dollars at some other institution in the province if they didn’t attend the institution in question? We’d argue these shouldn’t be counted in the impact figures, but in Canada many institutions choose to do so anyway (e.g., Calgary, UBC). Kudos to those which do not (e.g., McMaster, Lakehead, Nipissing). Definitely an example of competitive counting.
3. Exaggerating student expenditures (2). There is also the question of how much spending one attributes to those students. These documents are usually pretty cagey about data sources, but just within Ontario, we’ve seen monthly off-campus expenditure estimates of anything from $1,131.42 at Brock, which is probably low, to $1,949.27 at St. Lawrence college, which is probably too high (and yes, it is spurious precision to estimate these numbers to the penny). Probably, these reflect some kind of survey data collection effort. But the variation in Ontario is nothing like what you see in Australia, where the University of Wollongong pegs international student spending at about A$2,630, while the Group of 8 (i.e., the big research universities) assume monthly non-tuition spending of A$7,250, and then set the multiplier on that spending at a preposterous 3.14, because, who knows, maybe they like π or something.
4. Exaggerating visitor numbers. Most economic impact plans like to talk about the economic impact of visitors to campus. They are almost without exception based on fictional numbers, because nobody counts visitors. After all, practically speaking, how could you? If you’ve got a decent museum or gallery on campus (e.g., UBC), you can potentially do something around ticket sales. But the number of people who come to visit your students? It’s all entirely based on assumptions. Some universities assume 2 visitors per student, others assume 8. Brandon University, for some reason, uses a total number of visitors that works out to a figure of 23.6. The fancier ones make some differentiation between visits per international student vs. domestic ones, but again, it’s all competitive multiplication. Calgary seems to imply that everyone who comes to watch a Stampeders game is technically a U of C visitor (which is sort of true because U of C operates McMahon Stadium, but come on).
5. Exaggerating visitor revenues. Now we’re into real flight-of-fancy territory. If you don’t know how many visitors you actually have, you have no way of sampling them to find out how much a visitor would spend on an average visit. So normally what people do is just head over to the local tourist board to find out how much an average visitor spends. And voila! Multiply this number by the (largely fictional) number of visitors and there you go: tourism impact galore.

We could go on, but you get the idea.  Not all universities have the same relative impact: genuinely, some are better than others at generating it.  But given the standard methodologies used by the people who do economic impact assessments, the only obvious way to show an outsized impact, controlling for income, is through competitive counting and competitive multiplication.  And that probably gives the whole genre a bad name.

There are, however, better ways to talk about impact.  Ways that focus less on dollars and cents and more about the ways that institutions are embedded in communities and the paths by which benefits flow from institutions to society.  We’ve developed some innovative methods for this; give HESA Towers a shout if you’re interested in learning more.