Late this summer I came across a report from Polytechnics Canada, claiming to be a statement of Economic Impact of Polytechnics across the country. I have rarely read a document that left me with such profoundly mixed feelings, because it exemplified the absolute best and worst of the genre.

Long-time readers will know that I have a deep interest in the subject of institutional Economic Impact statements. Five years ago I wrote this piece explaining why I thought the standard practice of coming up with outlandish value figures (basically, treating costs as benefits and then finding a big fat multiplier) was not a credible way to sell universities and colleges. I then wrote a companion piece explaining how to make them better (tl;dr less quasi-economics, more narrative). About a year later I wrote a two-parter (here and here) examining how impact statements are made and how problematic some of the typical assumptions embedded in them can be. And then later, I showed exactly how an institution could craft its own quasi-economic economic impact statement on its own using just 14 numbers (most of them readily available within an institution) on the theory that if you absolutely must create a “big number” to impress government or donors or whoever, at least save some money and don’t go blowing it on KPMG or somebody that will charge you minimum $70K or so to come up with a figure which is really just competitive multiplication. 

If you know all that, I urge you to go ahead and read The Economic Impact of Applied Research at Canada’s Polytechnics. And then smile, and then weep.

Smile, because the report is a model of how to sell postsecondary education via stories and narrative. It provides case studies of applied research and then groups the cases in ways that show how applied research contributes to exporting, finding new markets, creating new jobs, etc. It’s not always done in the most artful way (the individual case study write-ups are cookie cutter and not especially compelling on their own), but collectively they are a powerful set of anecdotes and narratives that provide real examples of how these institutions contribute to the greater well-being.

(Personally, I think Polytechnics Canada may have missed a trick by not pointing out that in most cases applied research is really about providing a new or different set of learning experiences to produce better graduates, rather than about providing a “product” for an external partner. But that’s a quibble.)

That’s the good news. The problem is that it then goes on to make maybe the most dispiriting attempt to come up with a “big number” I have ever seen. I present it to you here in its entirety so that no one else ever makes this mistake again.

The model is lifted directly from a recent NBER paper by Benjamin Jones and Lawrence (“The Gender Gap is all about genetics”) Summer, entitled A Calculation of the Social Returns to Innovation. This model assumes that growth over the long run is a function of R&D expenditures. Not public R&D expenditures or applied public R&D expenditures—just R&D expenditures. Completely undifferentiated. From this, then, we can assume that every dollar of economic growth is a function of expenditures on R&D multiplied by the inverse of the social rate of return. The model then goes on to borrow social rates of return from two other sources—one from the Treasury Board of Canada (8%) and the other from a 2010 article in Canadian Public Policy (3.5%). Based on these rates, you get the result that $1 invested in R&D is worth between $8.09 and $18.49, a finding that Polytechnics Canada duly goes on to trumpet in its press release.

Now remember, this is not based on any analysis at all of what Canadian polytechnics do. It is a series of assumptions based on the idea that all R&D expenditure produces exactly the same benefit. That is, these returns would be the same whether the R&D is done by a polytechnic, a university, or a video game producer. It is not an argument specifically in favour of applied research at polytechnics.

But perhaps even more bizarre: it assumes per capita economic growth. We haven’t had any of that in Canada in about two and a half years now. Which of course is why the document chose to focus on the period 2016-19 when growth was positive. If you applied this exact same formula to the years 2022 to 2024, the result would be that the returns to research (including applied research at Polytechnics) were in fact negative. 

To be clear: I don’t believe (and neither should you) that the returns to research turn negative the instant per capita GDP growth does. But for precisely the same reason, I also don’t believe that every dollar in research gives out the bananas returns of 18:1. I mean, come on. Use some common sense. Research is not some kind of perpetual motion machine. If this were, it would be criminal for governments to spend money on anything but research.

I don’t want to come down too heavily on Polytechnics Canada here. If I had to guess, I would suspect that the folks there are perfectly aware that these numbers are nonsense. But likely they felt pressure—either from their members or from government—to come up with the “One Big Number” that would convince the folks at the Department of Finance that they’d be foolish not to invest in Polytechnics and applied research. But here’s the thing: there are some smart cookies at Finance. Folks who spend their professional lives studying cost-benefit returns like this. They are not going to be taken in by such obvious nonsense.

Plus, as I said, there is a lot of good stuff in this report. Good narrative explanations of impact they should have doubled down on and made more exciting and vivid, rather than trying to make a case based on some third-rate “Big Number” calculations which cannot have taken the consultants who produced the report more than a half-day to come up with (but for which I guarantee they charged a whole lot more). The latter, unfortunately, detracts from the former.

So, two cheers for Polytechnics Canada. But dear God, everyone: please stop with the Big Number nonsense. You’re making the whole sector stupider.