You may have seen the results of a poll out last week from Abacus Canada for Universities Canada, one which purports to look at “how Canadians feel about universities”.  I suspect you will hear a lot of this poll over the next few months, especially with respect to research and the Naylor report.  But it’s always worth approaching these things with a skeptical eye, so let’s spend a little time looking at the poll and how the questions were put together to see how much stock we should put in the answers.

Let’s start with this first set of questions:

The design of this first question is pretty good.   Respondents were offered two options and asked which one was closer to their opinion of Canadian universities.  Forced choices tend to give better responses than single answer questions, but the trick is in interpreting the response.  Some will probably try to tell you “73% of Canadians think universities are practical and up-to-date”, but that’s not quite right.  Rather, 73% of Canadians think they are more practical than impractical,and a similar percentage think they are closer to being up-to-date than to being out-of-date.

Much of the poll was devoted to research issue.  These next two questions are, to my mind, the most interesting.  The text is a little small, but the first question asks whether Canadians think research should focus on national competitiveness and the second questions asks whether the best way to lead in innovation is to invest in fundamental research.  The answer to both questions were exactly the same.

This is gold for Universities Canada.  “90% of Canadians agree that we need to invest in fundamental science in order to lead in innovation”.  But this would have been an interesting place to use one of those double-ended questions we saw at the start.  For instance: “Which is closer to your view: That Canada will gain leadership in innovation by investing in basic research or investing in applied research”?  The answers would have been instructive, but not nearly as politically useful to universities Canada.

Now, we get into the more dubiously worded questions. Take the questions where they ask about the importance of funding research. They aren’t just bad questions, they’re bad politics.

They are bad questions because you’re going to get all kinds of social acceptability bias creeping in.  Who is going to say they are against spending money on discovering new medicines?  And they’re potentially bad politics – from universities’ point of view anyway – because it assumes that university research has particular practical endpoints.  Some of it does of course.  But notice they didn’t ask about supporting research in the humanities. Ways of reconciling with Indigenous peoples, say.  Or ensuring no 18^th century Jesuit manuscript goes unread.  Wonder why.

(When reading a poll, always – always – look for the questions they don’t ask.)

Meanwhile the last set of questions, which are agree/disagree to a set of specific statements, is just shameful.

Look at that third one: “the government should spend more on universities because the upside for Canada is tremendous”.  I’ve heard of leading questions before but this is ridiculous.  Or the fourth one: “Canada has a chance to lead the world in higher education research and innovation”.  We have a chance to become the wine capital of the world, too.  Or World Cricket champions.  A chance, sure, but how much of one?  Meaningless term.

Now take a look at what questions Universities Canada didn’t ask.  They didn’t ask people how important any of these issues were, and where they ranked in their political priorities.  Sure, 86% say Ottawa should spend more on research.  All that tells us is that in the abstract, when no other priorities or financial constraints are in the picture, Canadians think supporting university research is a good idea.  But the real question is: where do they want to direct incremental government expenditure.  If Ottawa has a spare billion this year (and it’s increasingly looking like they will have more than that), where do they want it to go?  University research?  Or something else – like safe water in First Nations communities, cyberdefense, child care, deficit reduction, tax cuts?  And, more to the point, from a politician’s point of view, is this an issue that moves votes?  Is anyone going to change their vote if a government does or does not invest in this area?

Those are the real questions.  The fact that Universities Canada didn’t ask these question (or, possibly, asked them but didn’t publish them) suggests they know that clarity on this point probably wouldn’t help their case.