HESA

Higher Education Strategy Associates

Category Archives: Data

June 05

Student Health (Part 3)

You know how it is when someone tries to make a point about Canadian higher education using data from American universities? It’s annoying.  Makes you want to (verbally) smack them upside the head. Canada and the US are different, you want to yell. Don’t assume the data are the same! But of course the problem is there usually isn’t any Canadian data, which is part of why these generalizations get started in the first place.

Well, one of the neat things about the AHCA-NCHA campus health survey I was talking about last week is that it is one of the few data collection instruments that is in use on both sides of the border. Same questions, administered at the same time to tens of thousands of students on both sides of the border. And, as I started to look at the data for 2016, I realized my “Canada is different” rant is – with respect to students and health at least – almost entirely wrong. Turns out Canadian and American students are about as alike as two peas in a pod. It’s kind of amazing, actually.

Let’s start with basic some basic demographic indicators, like height and weight. I think I would have assumed automatically that American students would be both taller and heavier than Canadian ones, but figure 1 shows you what I know.

Figure 1: Median Height (Inches) and Weight (Pounds), Canadian vs. US students.

OTTSYD-1

Now, let’s move over to issues of mental health, one of the key topics of the survey. Again, we see essentially no difference between results on either side of the 49th parallel.

Figure 2: Within the last 12 months have you been diagnosed with/treated for…

OTTSYD-2

What about that major student complaint, stress? The AHCA-NCHA survey asks students to rate the stress they’ve been under over the past 12 months. Again, the patterns in the two countries are more or less the same.

Figure 3: Within the last 12 months, rate the stress you have been under.

OTTSYD-3

One interesting side-note here: students in both countries were asked about issues causing trauma or being “difficult to handle”. Financial matters were apparently more of an issue in Canada (40.4% saying yes) than in the US (33.7%). I will leave it to the reader to ponder how that result lines up with various claims about the effects of tuition fees.

At the extreme end of mental health issues, we have students who self-harm or attempt suicide. There was a bit of a difference on this one, but not much, with Canadian students slightly more likely to indicate that they had self-harmed or attempted suicide.

Figure 4: Attempts at Self-harm/suicide.

OTTSYD-4

What about use of tobacco, alcohol and illicit substances? Canadian students are marginally more likely to drink and smoke, but apart from that the numbers look pretty much the same. The survey, amazingly, does not ask about use of opioids/painkillers, which if books like Sam Quinones’ Dreamland are to be believed have made major inroads among America’s young – I’d have been interested to see the data on that. It does have a bunch of other minor drugs – heroin, MDMA, etc, and none of them really register in either country.

Figure 5: Use of Cigarettes, Alcohol, Marijuana, Cocaine.

OTTSYD-5

This post is getting a little graph-heavy, so let me just run through a bunch of topics where there’s essentially no difference between Canadians and Americans: frequency of sexual intercourse, number of sexual partners, use of most illegal drugs, use of seat belts, likelihood of being physically or sexually assaulted, rates of volunteering….in fact among the few places where you see significant differences between Canadian and American students is with respect to the kinds of physical ailments they report. Canadian students are significantly more likely to report having back pain, Americans more likely to report allergies and sinus problems.

Actually, the really big differences between the two countries were around housing and social life. In Canada, less than 2% of students reported being in a fraternity/sorority, compared to almost 10% in the United States. And as for housing, as you can see Americans are vastly more likely to live on-campus and vastly less-likely to live at home. On balance, that means they are incurring significantly higher costs to attend post-secondary education. Also, it probably means campus services are under a lot more pressure in the US than up here.

Figure 6: Student Living Arrangements.

OTTSYD-6

A final point here is with respect to perceptions of campus safety. We all know the differences in rates of violent crimes in the two countries, so you’d expect a difference in perceptions of safety, right? Well, only a little bit, only at night and mostly- off-campus. Figure 7 shows perceptions of safety during the day and at night, on campus and in the community surrounding campus.

Figure 7: Perceptions of safety on campus and in surrounding community.

OTTSYD-7

In conclusion: when it comes to students health and lifestyle, apart from housing there do not appear to many cross-border differences. We seem to be living in a genuinely continental student culture.

June 01

Student Health (part 1)

I have been perusing a quite astonishingly detailed survey that was recently released regarding student health.  Run by the American College Health Association-National College Health Assessment, this multi-campus exercise has been run twice now in Canada – once in 2013 and once in 2016.  Today, I’m going to look at what the 2016 results say, which are interesting in and of themselves.  Tomorrow, I’m going to look at how the data has changed since 2013 and why I think some claims about worsening student health outcomes (particularly mental health) need to be viewed with some caution.  If I get really nerdy over the weekend, I might do some Canada-US comparisons, too.

Anyways.

The 2016 study was conducted at 41 public institutions across Canada.  Because it’s an American based survey, it keeps referring to all institutions as “colleges”, which is annoying.  27 of the institutions are described as “4-year” institutions (which I think we can safely say are universities), 4 are described as “2-year” institutions (community colleges) and 10 described as “other” (not sure what to make of this, but my guess would be community colleges/polytechnics that offer mostly three-year programs).  In total, 43,780 surveys were filled out (19% response rate), with a roughly 70-30 female/male split.  That’s pretty common for campus surveys, but there’s no indication that responses have been re-weighted to match actual gender splits, which is a little odd but whatever.

 

There’s a lot of data here, so I’m mostly going to let the graphs do the talking.  First, the frequency of students with various disabilities.  I was a little bit surprised that psychiatric conditions and chronic illnesses were as high as they were.

Figure 1: Prevalence of Disabilities

Figure 1 Prevalence of Disabilities

Next, issues of physical safety.  Just over 87% of respondents reported feeling safe on campus during the daytime; however, only 37% (61% of women, 27% of men, and right away you can see how the gender re-weighting issue matters) say that they feel safe on campus at night.  To be fair, this is not a specific worry about campuses: when asked about their feelings of personal safety in the surrounding community, the corresponding figures were 62% and 22%.  Students were also asked about their experiences with specific forms of violence over the past 12 months.  As one might imagine, most of the results were fairly highly gendered.

 

Figure 2: Experience of Specific Forms of Violence Over Past 12 Months, by Gender

Figure 2 Experience of Specific Forms of Violence

Next, alcohol, tobacco, and marijuana.  This was an interesting question as the survey not only asked students about their own use of these substances, but also about their perception of other students’ use of them.  It turns out students vastly over-estimate the number of other students who engage with these substances.  For instance, only 11% of students smoked cigarettes in the past 30 days (plus another 4% using e-cigarettes and 3% using hookahs), but students believed that nearly 80% of students had smoked in the past month.

 

Figure 3: Real and Perceived Incidence of Smoking, Drinking and Marijuana Use over Past 30 Days

Figure 3 Real and Perceived Incidence of Smoking

Figure 4 shows the most common conditions for which students had been diagnosed with and/or received treatment for in the last twelve months.  Three of the top ten and two of the top three were mental health conditions.

Figure 4: Most Common Conditions Diagnosed/Treated in last 12 Months

Figure 4 Most Common Conditions Diagnosed

Students were also asked separately about the kinds of things that had negatively affected their academics over the previous year (defined as something which had resulted in a lower mark than they would have otherwise received).  Mental health complaints are very high on this list; much higher in fact than actual diagnoses of such conditions.  Also of note here: internet gaming was sixth among factors causing poorer marks; finances only barely snuck into the top 10 reasons, with 10.3% citing it (though elsewhere in the study over 40% said they had experienced stress or anxiety as a result of finances).

Figure 5: Most Common Conditions Cited as Having a Negative Impact on Academics

Figure 5 Most Common Conditions Cited as Having

A final, disturbing point here: 8.7% of respondents said they had intentionally self-harmed over the past twelve months, 13% had seriously contemplated suicide and 2.1% said they had actually attempted suicide.  Sobering stuff.

April 05

Student/Graduate Survey Data

This is my last thought on data for awhile, I promise.  But I want to talk a little bit today about what we’re doing wrong with the increasing misuse of student and graduate surveys.

Back about 15 years ago, the relevant technology for email surveys became sufficiently cheap and ubiquitous that everyone started using them.  I mean, everyone.  So what has happened over the last decade and a half has been a proliferation of surveys and with it – surprise, surprise – a steady decline in survey response rates.  We know that these low-participation surveys (nearly all are below 50%, and most are below 35%) are reliable, in the sense that they give us similar results year after year.  But we have no idea whether they are accurate, because we have no way of dealing with response bias.

Now, every once in awhile you get someone with the cockamamie idea that the way to deal with low response rates is to expand the sample.  Remember how we all laughed at Tony Clement when he claimed  the (voluntary) National Household Survey would be better than the (mandatory) Long-Form Census because the sample size would be larger?  Fun times.  But this is effectively what governments do when they decide – as the Ontario government did in the case of its sexual assault survey  – to carry out what amounts to a (voluntary) student census.

So we have a problem: even as we want to make policy on a more data-informed basis, we face the problem that the quality of student data is decreasing (this also goes for graduate surveys, but I’ll come back to those in a second).  Fortunately, there is an answer to this problem: interview fewer students, but pay them.

What every institution should do – and frankly what every government should do as well – is create a balanced, stratified panel of about 1000 students.   And it should pay them maybe $10/survey to complete surveys throughout the year.  That way, you’d have good response rates from a panel that actually represented the student body well, as opposed to the crapshoot which currently reigns.  Want accurate data on student satisfaction, library/IT usage, incidence of sexual assault/harassment?  This is the way to do it.  And you’d also be doing the rest of your student body a favour by not spamming them with questionnaires they don’t want.

(Costly?  Yes.  Good data ain’t free.  Institutions that care about good data will suck it up).

It’s a slightly different story for graduate surveys.  Here, you also have a problem of response rates, but with the caveat that at least as far as employment and income data is concerned, we aren’t going to have that problem for much longer.  You may be aware of Ross Finnie’s work  linking student data to tax data to work out long-term income paths.  An increasing number of institutions are now doing this, as indeed is Statistic Canada for future versions of its National Graduate Survey (I give Statscan hell, deservedly, but for this they deserve kudos).

So now that we’re going to have excellent, up-to-date data about employment and income data we can re-orient our whole approach to graduate surveys.  We can move away from attempted censuses with a couple of not totally convincing questions about employment and re-shape them into what they should be: much more qualitative explorations of graduate pathways.  Give me a stratified sample of 2000 graduates explaining in detail how they went from being a student to having a career (or not) three years later rather than asking 50,000 students a closed-ended question about whether their job is “related” to their education every day of the week.  The latter is a boring box-checking exercise: the former offers the potential for real understanding and improvement.

(And yeah, again: pay your survey respondents for their time.  The American Department of Education does it on their surveys and they get great data.)

Bottom line: We need to get serious about ending the Tony Clement-icization of student/graduate data. That means getting serious about constructing better samples, incentivizing participation, and asking better questions (particularly of graduates).  And there’s no time like the present. If anyone wants to get serious about this discussion, let me know: I’d be overjoyed to help.

April 04

How to Think about “Better Higher Education Data”

Like many people, I am in favour of better data on the higher education sector.  But while this call unites a lot of people, there is remarkably little thinking that goes into the question of how to achieve it.  This is a problem, because unless we arrive at a better common understanding of both the cost and the utility of different kinds of data, we are going to remain stuck in our current position.

First, we need to ask ourselves what data we need to know versus what kinds of data it would be nice to know.  This is of course, not a value-free debate: people can have legitimate differences about what data is needed and what is not.  But I think a simple way to at least address this problem is to ask of any proposed data collection: i) what questions does this data answer?  And ii) what would we do differently if we knew the answer to that question?  If the answer to either question is vague, maybe we should put less emphasis on the data.

In fact, I’d argue that most of the data that institutions and government are keen to push out are pretty low on the “what would we do differently” scale.  Enrolments (broken down in various ways), funding, etc – those are all inputs.  We have all that data, and they’re important: but they don’t tell you much about what’s going right or going wrong in the system.  They tell you what kind of car you’re driving, but not your speed or direction.

What’s the data we need?  Outputs.  Completion rates.  Transitions rates.  At the program, institutional and system level. Also outcomes: what happens to graduates?  How quickly do they transition to permanent careers. And do they feel their educational career was a help or a hindrance to getting the career they wanted?  And yeah, by institution and (within reason) by program.  We have some of this data, in some parts of the country (BC is by far the best at this) but even here we rely far too heavily on some fairly clumsy quantitative indicators and not enough on qualitative information like: “what do graduates three years out think the most/least beneficial part of their program was?”

Same thing on research.  We need better data on PhD outcomes.  We need a better sense of the pros and cons of more/smaller grants versus fewer/larger ones.  We need a better sense of how knowledge is actually transferred from institutions to firms, and what firms do with the knowledge in terms of turning them into product or process innovations.  Or, arguably, on community impact (though there I’m not completely convinced we even know yet what the right questions are).

Very few of these questions can be answered through big national statistical datasets about higher education.  Even when it comes to questions like access to education, it’s probably far more important that we have more data on people who do not go to PSE than to have better or speedier data access to enrolment data.  And yet we have nothing on this, and haven’t had since the Youth in Transition Survey ended.  But that’s expensive and could cost upwards of ten millions of dollars.  Smaller scale, local studies could be done for a fraction of the cost – if someone were willing to fund them.

There are actually enormous data resources available at the provincial and institutional level to work from.  Want to find individuals who finished high school but didn’t attend PSE?  Most provinces now have individual student numbers which could be used to identify these individuals and bring them into a study.  Want to look at program completion rates?  All institutions have the necessary data: they just choose not to release it.

All of which is to say: we could have a data revolution in this country.  But it’s not going to come primarily from better national data sets run by Statistics Canada.  It’s going to come from lots of little studies creating a more data-rich environment for decision-making.  It’s going to come from governments and institutions changing their data mindsets from one where hoarding data is the default to one where publishing is the default.  It’s going to come from switching focus from inputs to outputs.

Even more succinctly: we are all part of the problem.  We are all part of the solution.  Stop waiting for someone else to come along and fix it.

April 03

Data on Race/Ethnicity

A couple of week ago, CBC decided to make a big deal about how terrible Canadian universities were for not collecting data on race (see Why so many Canadian universities Know so little about their own racial diversity). As you all know, I’m a big proponent of better data in higher education. But the effort involved in getting new data has to be in some way proportional to the benefit derived from that data. And I’m pretty sure this doesn’t meet that test.

In higher education, there are only two points where it is easy to collect data from students: at the point of application, and at the point of enrolment. But here’s what the Ontario Human Rights Code has to say about collecting data on race/ethnicity in application forms:

Section 23(2) of the Code prohibits the use of any application form or written or oral inquiry that directly or indirectly classifies an applicant as being a member of a group that is protected from discrimination. Application forms should not have questions that ask directly or indirectly about race, ancestry, place of origin, colour, ethnic origin, citizenship, creed, sex, sexual orientation, record of offences, age, marital status, family status or disability.

In other words, it’s 100% verboten. Somehow, CBC seems to have missed this bit. Similar provisions apply to data collected at the time of enrolment –a school still needs to prove that there is a bona fide reason related to one’s schooling in order to require a student to answer the question. So generally speaking, no one asks a question at that point either.

Now, if institutions can’t collect relevant data via administrative means, what they have to do to get data on race/ethnicity is move to a voluntary survey. Which in fact they do, regularly. Some do a voluntary follow-up survey of applicants through Academica, others attach race/ethnicity questions on the Canadian Undergraduate Survey Consortium (CUSC) surveys, others attach it to NSSE. Response rates on these surveys are not great: NSSE sometimes gets 50% but that’s the highest rate available. And, broadly speaking, they get high-level data about their student body. The data isn’t great quality because of the response rate isn’t fabulous and the small numbers mean that you can’t really subdivide ethnicity very much (don’t expect good numbers on Sikhs v. Tamils), but one can know at a rough order of magnitude what percentage of the student body is visible minority, what percentage self-identifies as aboriginal, etc. I showed this data at a national level back here.

Is it possible to get better data? It’s hard to imagine, frankly. On the whole, students aren’t crazy about being surveyed all the time. NSSE has the highest response rate of any survey out there, and CUSC isn’t terrible either (though it tends to work on a smaller sample size). Maybe we could ask slightly better questions about ethnicities, maybe we could harmonize the questions across the two surveys. That could get you data at institutions which cover 90% of institutions in English Canada (at least).

Why would we want more than that? We already put so much effort into these surveys: why go to all kinds of trouble to do a separate data collection activity which in all likelihood would have worse response rates than what we already have?

It would be one thing, I think, if we thought Canadian universities had a real problem in not admitting minority students. But the evidence at the moment the opposite: that visible minority students in fact attend at a rate substantially higher than their share of the population. It’s possible of course that some sub-sections of the population are not doing as well (the last time I looked at this data closely was a decade ago, but youth from the Caribbean were not doing well at the time). But spending untold dollars and effort to get at that problem in institutions across country when really the Caribbean community in Canada is clustered in just two cities (three, if you count the African Nova Scotians in Halifax)? I can’t see it.

Basically, this is one of those cases where people are playing data “gotcha”. We actually do know (more or less) where we are doing well or poorly at a national level. On the whole, where visible minorities are concerned, we are doing well. Indigenous students? Caribbean students? That’s a different story. But we probably don’t need detailed institutional data collection to tell us that. If that’s really what the issue is, let’s just deal with it. Whinging about data collection is just a distraction.

March 27

Losing Count

Stop me if you’ve heard this story before: Canada is not sufficiently innovative, and part of the reason is that we don’t spend enough on research.  It’s not that we don’t spend enough on *public* research; adjusted for GDP, we actually do above-average on that.  What pulls us down is in international comparisons corporate R & D.  Our narrow-minded, short-sighted, resource-obsessed business class spends far less on R&D than its equivalents in most other country, and that is what gives us such a low overall R&D spend.

Familiar?  It should be; it’s been standard cant in Canada for a couple of decades at least.  And it gets used to argue for two very specific things.  There’s the argument which basically says “look, if private R&D is terrible, we’ll just have to make it up on the public side, won’t we?”, and where else to spend but on university research?  (Universities Canada used to make this argument quite a bit, but not so much lately AFAIK).  Then there’s the argument that says: well, since under the linear model of innovation in which public “R” leads to private “D”, the problem must be that public “R” is too theoretical on insufficiently focussed on areas of national industrial strength – and what we really need to do is make research more applied/translational/whatever.

But what if that story is wrong?

Last year, the Impact Centre at the University of Toronto put out a little-noticed paper called Losing Count. It noted a major problem related to the collection and reporting of R&D.  Starting in 1997, Statistics Canada adopted a definition of Research and Development which aligned with Canada’s tax laws.  This makes perfect sense from a reporting point of view, because it reduces the reporting burden on big corporations (they can use the same data twice).  But from a measuring Canada against other countries perspective, it’s not so good, because it means the Canadian statistics are different from those in the rest of the world.

Specifically, Canada since 1997 has under-reported Business R&D in two ways.  First, it does not report any R&D in the social sciences and humanities.  All those other OECD countries are reporting research in business, financial management, psychology, information science, etc., but we are not.  Second, work that develops or improves materials, products and processes, but that draws on existing knowledge rather than new scientific or new technological advances is not counted as Research & Development in Canada but is counted elsewhere.

How big a problem is this?  Well, one problem is that literally every time the Canada Revenue Agency tightens eligibility for tax credits, reported business R&D falls.  As this has happened a number of times over the past two decades, it may well be that our declining business R&D figures are actually a function of stricter tax laws than they are of changing business activity.  As for the difference in absolute amount being measured, it’s impossible to say.  The authors of the study took a sample of ten companies (which they recognize as not being scientific in any way) and determined that if the broader, more OECD-consistent definition were used, spending on R&D salaries would rise by a factor of three.  If that were true across the board (it probably isn’t) it would shift Canada from being one of the world’s weakest business R&D performers to one of the best.

Still, even if this particular result is not generalizable, the study remains valuable for two reasons.  First, it underlines how tough it is for statistical agencies to capture data on something as fluid and amorphous as research and development in a sensible, simple way.  And second, precisely because data is so hard to collect, international comparisons are extremely hard to make.  National data can be off by a very wide factor simply because statistical agencies make slightly different decisions about to collect data efficiently.

The takeaway is this:  the next time someone tells a story about how innovation is being throttled by lack of business spending on research (compared to say, the US or Sweden), ask them if they’ve read Losing Ground.  Because while this study isn’t the last word on the subject, it poses questions that no one even vaguely serious about playing in the Innovation space should be able to ignore.

February 23

Garbage Data on Sexual Assaults

I am going to do something today which I expect will not put me in good stead with one of my biggest clients.  But the Government of Ontario is considering something unwise and I feel it best to speak up.

As many of you know, the current Liberal government is very concerned about sexual harassment and sexual assault on campus, and has devoted no small amount of time and political capital to getting institutions to adopt new rules and regulations around said issues.  One can doubt the likely effectiveness of such policies, but not the sincerity of the motive behind them.

One of the tools the Government of Ontario wishes to use in this fight is more public disclosure about sexual assault.  I imagine they have been influenced with how the US federal government collects and publishes statistics on campus crime, including statistics on sexual assaults.  If you want to hold institutions accountable for making campuses safer, you want to be able to measure incidents and show change over time, right?

Well, sort of.  This is tricky stuff.

Let’s assume you had perfect data on sexual assaults by campus.  What would that show?  It would depend in part on the definitions used.  Are we counting sexual assaults/harassment which occur on campus?  Or are we talking about sexual assaults/harassment experiences by students?  Those are two completely different figures.  If the purpose of these figures is accountability and giving prospective students the “right to know” (personal safety is after all a significant concern for prospective students), how useful is that first number?  To what extent does it make sense for institutions to be held accountable for things which do not occur on their property?

And that’s assuming perfect data, which really doesn’t exist.  The problems multiply exponentially when you decided to rely on sub-standard data.  And according to a recent Request for Proposals placed on the government tenders website MERX, the Government of Ontario is planning to rely on some truly awful data for its future work on this file.

Here’s the scoop: the Ministry of Advanced Education and Skills Development is planning to do two surveys: one in 2018 and one in 2024.  They plan on getting contact lists of emails of every single student in the system – at all 20 public universities, 24 colleges and 417 private institutions and handing them over to a contractor so they can do a survey. (This is insane from a privacy perspective – the much safer way to do this is to get institutions to send out an email to students with a link to a survey so the contractor never sees the names without students’ consent).  Then they are going to send out an email to all those students – close to 700,000 in total – offering $5/per head to answer a survey.

Its not clear what Ontario plans to do with this data.  But the fact that they are insistent that *every* student at *every* institution be sent the survey suggests to me that they want the option to be able to analyze and perhaps publish the data from this anonymous voluntary survey on a campus by campus basis.

Yes, really.

Now, one might argue: so what?  Pretty much every student survey works this way.  You send out a message to as many students as you can, offer an inducement and hope for the best in terms of response rate.  Absent institutional follow-up emails, this approach probably gets you a response rate between 10 and 15% (a $5 incentive won’t move that many students)  Serious methodologists grind their teeth over those kinds of low numbers, but increasingly this is the way of the world.  Phone polls don’t get much better than this.  The surveys we used to do for the Globe and Mail’s Canadian University Report were in that range.  The Canadian University Survey Consortium does a bit better than that because of multiple follow-ups and strong institutional engagement.  But hell, even StatsCan is down to a 50% response rate on the National Graduates Survey.

Is there non-response bias?  Sure.  And we have no idea what it is.  No one’s ever checked.  But these surveys are super-reliable even if they’re not completely valid.  Year after year we see stable patterns of responses, and there’s no reason to suspect that the non-response bias is different across institutions.  So if we see differences in satisfaction of ten or fifteen percent from one institution from another, most of us in the field are content to accept that finding.

So why is the Ministry’s approach so crazy when it’s just using the same one as every one else?  First of all, the stakes are completely different.  It’s one thing to be named an institution with low levels of student satisfaction.  It’s something completely different to be called the sexual assault capital of Ontario.  So accuracy matters a lot more.

Second, the differences between institutions are likely to be tiny.  We have no reason to believe a priori that rates differ much by institutions.  Therefore small biases in response patterns might alter the league table (and let’s be honest, even if Ontario doesn’t publish this as a league table, it will take the Star and the Globe about 30 seconds to turn it into one).  But we have no idea what the response biases might be and the government’s methodology makes no attempt to work that out.

Might people who have been assaulted be more likely to answer than those who did not?  If so, you’re going to get inflated numbers.  Might people have reasons to distort the results?  Might a Men’s Rights group encourage all its members to indicate they’d been assaulted to show that assault isn’t really a women’s issue?  With low response rates, it wouldn’t take many respondents to get that tactic to work.

The Government is never going to get accurate overall response rates from this approach.  They might, after repeated tries, start to see patterns in the data: sexual assault is more prevalent in institutions in large communities than in small ones, maybe; or it might happen more often to students in certain fields of study than others.  That might be valuable.  But if the first time the data is published all that makes the papers is a rank order of places where students are assaulted, we will have absolutely no way to contextualize the data, no way to assess its reliability or validity.

At best, if the data is reported system-wide, the data will be weak.  A better alternative would be to go with a smaller random sample and better incentives so as to obtain higher  response rates.  But if it remains a voluntary survey *and* there is some intention to publish on a campus-by campus basis, then it will be garbage.  And garbage data is a terrible way to support good policy objectives.

Someone – preferably with a better understanding of survey methodology – needs to put a stop to this idea.  Now.

January 27

A Slice of Canadian Higher Education History

There are a few gems scattered through Statistics Canada’s archives. Digging around their site the other day, I came across a fantastic trove of documents published by the Dominion Bureau of Statistics (as StatsCan used to be called) called Higher Education in Canada. The earliest number in this series dates from 1938, and is available here. I urge you to read the whole thing, because it’s a hoot. But let me just focus in on a couple of points in this document worth pondering.

The first point of interest is the enrolment statistics (see page 65 of the PDF, 63 of the document). It won’t of course surprise anyone to know that enrolment at universities was a lot smaller in 1937-38 than it is today (33,600 undergraduates then, 970,000 or so now), or that colleges were non-existent back then. What is a bit striking is the large number of students being taught in universities who were “pre-matriculation” (i.e. high school students). Nearly one-third of all students in universities in 1937-38 had this “pre-matric” status. Now, two-thirds of these were in Quebec, where the “colleges classiques” tended to blur the line between secondary and post-secondary (and, in their new guise as CEGEPs, still kind of do). But outside of British Columbia, all universities had at least some pre-matric, which would have made these institutions quite different from modern ones.

The second point of interest is the section on entrance requirements at various universities (page 12-13 of the PDF, p. 10-11 of the document). With the exception of UNB, every Canadian university east of the Ottawa River required Latin or Greek in order to enter university, as did Queens, Western and McMaster. Elsewhere, Latin was an alternative to Mathematics (U of T), or an alternative to a modern language (usually French or German). What’s interesting here is not so much the decline in salience of classical languages, but the decline in salience of any foreign language. In 1938, it was impossible to gain admission to a Canadian university without first matriculating in a second language, and at a majority of them a third language was required as well. I hear a lot of blah blah about internationalization on Canadian campuses, but 80 years on there are no Canadian universities which require graduates to learn a second language, let alone set this as a condition of entry. An area, clearly, where we have gone backwards.

The third and final bit to enjoy is the section on tuition fees (page 13), which I reproduce here:

ottsyd-20170126

*$1 in 1937-38 = $13.95 in 2016
**$1 in 1928-29 = $16.26 in 2016

Be a bit careful in comparing across years here: because of deflation, $100 in 1928 was worth $85 in 1937 and so institutions which kept prices stable in fact saw a rise in income in real terms. There are a bunch of interesting stories here, including the fact that institutions had very different pricing strategies in the depression. Some (e.g. McGill, Saskatchewan, Acadia) increased tuition while others (mostly Catholic institutions like the Quebec seminaries and St. Dunstan’s) either held the line or reduced costs. Also mildly amusing is the fact that McGill’s tuition for in-province students is almost unchanged since 1937-38 (one can imagine the slogan: “McGill – we’ve been this cheap since the Rape of Nanking!”).

The more interesting point here is that if you go back to the 1920s, not all Canadian universities were receiving stable and recurrent operating grants from provincial governments (of note: nowhere in this digest of university statistics is government funding even mentioned). Nationally, in 1935, all universities combined received $5.4 million from provincial governments – and U of T accounted for about a quarter of that. For every dollar in fees universities received from students, they received $1.22 from government. So when you see that universities were for the most part charging around $125 per students in 1937-38, what that means is that total operating funding per student was maybe $275, or a shade under $4500 per student in today’s dollars. That’s about one-fifth of today’s operating income per student.

While most of that extra per-student income has gone towards making institutions more capital-intensive (scientific facilities in general were pretty scarce in the 1930s), there’s no question that the financial position of academics had improved. If you take a quick gander at page 15, which shows the distribution of professorial salaries, you’ll see that average annual salaries for associate profs was just below $3500, while those for full professors was probably in the $4200 range. Even after for inflation, that means academic salaries were less than half what they are today. Indeed, one of the reasons tenure was so valued back then was that job security made up for the not-stellar pay. Times change.

In any case, explore this document on your own: many hours (well, minutes anyway) of fun to be had here.

January 18

More Bleak Data, But This Time on Colleges

Everyone seems to be enjoying data on graduate outcomes, so I thought I’d keep the party going by looking at similar data from Ontario colleges. But first, some of you have written to me suggesting I should throw some caveats on what’s been covered so far. So let me get a few things out of the way.

First, I goofed when saying that there was no data on response rates from these surveys. Apparently there is and I just missed it. The rate this year was 40.1%, a figure which will make all the economists roll their eyes and start muttering about response bias, but which anyone with field experience in surveys will tell you is a pretty good response for a mail survey these days (and since the NGS response rate is now down around the 50% mark, it’s not that far off the national “gold standard”).

Second: all this data on incomes I’ve been giving you is a little less precise than it sounds. Technically, the Ontario surveys do not ask income, they ask income ranges (e.g. $0-20K, $20-40K, etc). When data is published either by universities or the colleges, this is turned into more precise-looking figures by assigning the mid-point value of each and then averaging those points. Yes, yes, kinda dreadful. Why can’t we just link this stuff to tax data like EPRI does? Anyways, that means you should probably take the point values with a pinch of salt: but the trend lines are likely still meaningful.

Ok, with all that out of the way, let’ turn to the issue of colleges. Unfortunately, Ontario does not collect or display data on college graduates’ outcomes the way they do for universities. There is no data around income, for instance. And no data on employment 2 years after graduation, either. The only real point of comparison is employment 6 months after graduation, and even this is kind of painful: for universities the data is available only by field of study; for colleges, it is only available by institution. (I know, right?) And even then it is not even calculated on quite the same basis: universities include graduates with job offers while the college one does not. So you can’t even quite do an apples-to-apples comparison, even at the level of the sector as a whole. But if you ignore that last small difference in calculation and focus not on the point-estimates but on the trends, you can still see something interesting. Here we go:

Figure 1: Employment Rates 6 months after Graduation, Ontario Universities vs. Ontario Colleges, by Graduating Cohort, 1999-2015

ottsyd-20170117

So, like I said, ignore the actual values in Figure 1 because they’re calculated in two slightly different ways; instead, focus on the trends. And if you do that, what you see is (a blip in 2015 apart), the relationship between employment rates in the college and university sector looks pretty much the same throughout the period. Both had a wobble in the early 2000s, and then both took a big hit in the 2008 recession. Indeed, on the basis of this data, it’s hard to make a case that one sector has done better than another through the latest recession: both got creamed, neither has yet to recover.

(side point: why does the university line stop at 2013 while the college one goes out to 2015? Because Ontario doesn’t interview university grads until 2 years after grad and then asks them retroactively what they were doing 18 months earlier. So the 2014 cohort was just interviewed last fall and it’ll be a few months until their data is released. College grads *only* get interviewed at 6 months, so data is out much more quickly)

What this actually goes is put a big dent in the argument that the problem for youth employment is out-of-touch educators, changing skill profiles, sociologists v. welders and all that other tosh people were talking a few years ago. We’re just having more trouble than we used to integrating graduates into the labour market. And I’ll be taking a broader look at that using Labour Force Survey data tomorrow.

January 13

Restore the NGS!

One of the best things that Statistics Canada ever did in the higher education field was the National Graduates’ Survey (NGS). OK, it wasn’t entirely Statscan – NGS has never been a core product funded from the Statscan budget but rather funded periodically by Employment and Social Development Canada (ESDC) or HRDC or HRSDC or whatever earlier version of the department you care to name – but they were the ones doing the execution. After a trial run in the late 1970s (the results of which I outlined back here), Statscan tracked the results of the graduating cohorts of 1982, 1986, 1990, 1995, 2000 and 2005 two and five years after graduation (technically, only the 2-year was called NGS – the 5-year survey was called the Follow-up of Graduates or FOG but no one used the name because it was too goofy). It became the prime way Canada tracked transitions from post-secondary education to the labour market, and also issues related to student debt.

Now NGS was never a perfect instrument. Most of the income data could have been obtained much more simply through administrative records, the way Ross Finnie is currently doing at EPRI. We could get better data on student debt of provinces ever got their act together and actually released student data on a consistent and regular basis (I’m told there is some chance of this happening in the near future). It didn’t ask enough questions about activities in school, and so couldn’t examine the effects of differences in provision (except for, say, Field of Study) on later outcomes. But for all that it was still a decent survey, and more to the point one with a long history which allowed one to make solid comparisons over time.

Then, along comes budget cutting exercises during the Harper Government. ESDC decides it only has enough money for one survey, not two. Had Statscan or ESDC bothered to consult anyone about what to do in this situation, the answer would almost certainly have been: keep the 2-year survey and ditch the 5-year one. The 5-year survey was always beset with the twin problems of iffy response rates and being instantly out of date by the time it came out (“that was seven graduating classes ago!” people would say – “what about today’s graduates”?). But the 2-year? That was gold, with a decent time series going back (in some topic areas) back almost 30 years. Don’t touch that, we all would have said, FOR GOD’S SAKE DON’T TOUCH IT, LEAVE IT AS IT IS.

But of course, Statscan and ESDC didn’t consult and they didn’t leave it alone. Instead of sticking with a 2-years out survey, they decided to do a survey of students three years out, thereby making the results for labour market transitions totally incompatible with the previous six iterations of the survey. They spent millions to get a whole bunch of data which was hugely sub-optimal because they murdered a perfectly good time-series to get it.

I have never heard a satisfactory explanation as to why this happened. I think it’s either a) someone said: “hey, if we’re ditching a 2-year and a 5-year survey, why not compromise and make a single 3-year survey?” or b) Statscan drew a sample frame from institutions for the 2010 graduating class, ESDC held up the funding until it was too late to do a two-year sample and then when it eventually came through Statscan said, “well we already have a frame for 2010, so why not sample them three years out instead of doing the sensible thing and going back and getting a new frame for the 2011 cohort which would allow us to sample two years out”. To be clear, both of these possible reasons are ludicrous and utterly indefensible as a way to proceed with a valuable dataset, albeit in different ways. But this is Ottawa so anything is possible.

I have yet to hear anything about what, if anything, Statscan and ESDC plan to do about surveying the graduating cohort of 2015. If they were going to return to a two-year cycle, that would mean surveying would have to happen this spring; if they’re planning on sticking with three, the survey would happen in Spring 2018. But here’s my modest proposal: there is nothing more important about NGS than bringing back the 2-year survey frame. Nothing at all. Whatever it takes, do it two years out. If that means surveying the class of 2016 instead of 2015, do it. We’ll forget the Class of 2010 survey ever happened. Do not, under any circumstances, try to build a new standard based on a 3-year frame. We spent 30 years building a good time series at 24 months out from graduation. Better have a one-cycle gap in that time series than spend another 30 years building up an equally good time-series at 36 months from graduation.

Please, Statscan. Don’t mess this up.

Page 1 of 1512345...10...Last »