HESA

Higher Education Strategy Associates

Tag Archives: Graduate Outcomes

April 05

Student/Graduate Survey Data

This is my last thought on data for awhile, I promise.  But I want to talk a little bit today about what we’re doing wrong with the increasing misuse of student and graduate surveys.

Back about 15 years ago, the relevant technology for email surveys became sufficiently cheap and ubiquitous that everyone started using them.  I mean, everyone.  So what has happened over the last decade and a half has been a proliferation of surveys and with it – surprise, surprise – a steady decline in survey response rates.  We know that these low-participation surveys (nearly all are below 50%, and most are below 35%) are reliable, in the sense that they give us similar results year after year.  But we have no idea whether they are accurate, because we have no way of dealing with response bias.

Now, every once in awhile you get someone with the cockamamie idea that the way to deal with low response rates is to expand the sample.  Remember how we all laughed at Tony Clement when he claimed  the (voluntary) National Household Survey would be better than the (mandatory) Long-Form Census because the sample size would be larger?  Fun times.  But this is effectively what governments do when they decide – as the Ontario government did in the case of its sexual assault survey  – to carry out what amounts to a (voluntary) student census.

So we have a problem: even as we want to make policy on a more data-informed basis, we face the problem that the quality of student data is decreasing (this also goes for graduate surveys, but I’ll come back to those in a second).  Fortunately, there is an answer to this problem: interview fewer students, but pay them.

What every institution should do – and frankly what every government should do as well – is create a balanced, stratified panel of about 1000 students.   And it should pay them maybe $10/survey to complete surveys throughout the year.  That way, you’d have good response rates from a panel that actually represented the student body well, as opposed to the crapshoot which currently reigns.  Want accurate data on student satisfaction, library/IT usage, incidence of sexual assault/harassment?  This is the way to do it.  And you’d also be doing the rest of your student body a favour by not spamming them with questionnaires they don’t want.

(Costly?  Yes.  Good data ain’t free.  Institutions that care about good data will suck it up).

It’s a slightly different story for graduate surveys.  Here, you also have a problem of response rates, but with the caveat that at least as far as employment and income data is concerned, we aren’t going to have that problem for much longer.  You may be aware of Ross Finnie’s work  linking student data to tax data to work out long-term income paths.  An increasing number of institutions are now doing this, as indeed is Statistic Canada for future versions of its National Graduate Survey (I give Statscan hell, deservedly, but for this they deserve kudos).

So now that we’re going to have excellent, up-to-date data about employment and income data we can re-orient our whole approach to graduate surveys.  We can move away from attempted censuses with a couple of not totally convincing questions about employment and re-shape them into what they should be: much more qualitative explorations of graduate pathways.  Give me a stratified sample of 2000 graduates explaining in detail how they went from being a student to having a career (or not) three years later rather than asking 50,000 students a closed-ended question about whether their job is “related” to their education every day of the week.  The latter is a boring box-checking exercise: the former offers the potential for real understanding and improvement.

(And yeah, again: pay your survey respondents for their time.  The American Department of Education does it on their surveys and they get great data.)

Bottom line: We need to get serious about ending the Tony Clement-icization of student/graduate data. That means getting serious about constructing better samples, incentivizing participation, and asking better questions (particularly of graduates).  And there’s no time like the present. If anyone wants to get serious about this discussion, let me know: I’d be overjoyed to help.

January 17

Another Lens on Bleak Graduate Income Data

So, yesterday we looked at Ontario university graduate employment data (link to: previous).  Today I want to zero in a little bit on what’s happening by field of study.

(I can hear two objections popping up already.  First; “why just Ontario”?  Answer: while Quebec, Alberta, British Columbia and the Maritimes – via MPHEC – all publish similar data, they all publish the data in slightly different ways, making it irritating (and in some cases impossible) to come up with a composite national figure.  The National Graduate Survey (NGS) in theory does this, but only every five years but as I explained last week has made itself irrelevant by changing the survey period.  So, in short, I can’t do national, and Ontario a) is nearly half the country in terms of university enrolments and b) publishes slightly more detailed data than most.  Second, “why just universities”?  Answer: “fair point, I’ll be publishing that data soon”.

Everyone clear? OK, let’s keep going).

Let’s look first at employment rates 6 months after graduation by field of study (I include only the six largest – Business/Commerce, Education, Engineering, Humanities, Physical Sciences and Social Sciences – because otherwise these graphs would be an utter mess), shown below in Figure 1.  As was the case yesterday, the dates along the x-axis are the cohort graduation year.

ottsyd-20170116-1

Two take-aways here, I think.  The first is that the post-08 recession really affected graduates of all fields more or less equally, with employment rates falling by between 6 and 8 percentage points (the exception is humanities, where current rates are only four percentage points below where they were in 2007).  The second is that pretty much since 2001, it’s graduates in the physical sciences who have had the weakest results.

OK, but as many in the academy say: 6 months isn’t enough to judge anything.  What about employment rates after, say, 2 years?  These are shown below in Figure 2.

ottsyd-20170116-2

This graph is smoother than the previous one, which suggests the market for graduates with 2 years in the labour market is a lot more stable than that for graduates with just 6 months.    If you compare the class of 2013 with the clss of 2005 (the last one to completely miss the 2008-9 recession), business and commerce students’ employment rates have fallen only by one percentage point while those in social sciences have dropped by six percentage points, with the others falling somewhere in between.  One definite point to note for all those STEM enthusiasts out there: there’s no evidence here that students in STEM programs have fared much better than everyone else.

But employment is one thing; income is another.  I’ll spare you the graph of income at six months because really, who cares?  I’ll just go straight to what’s happening at two years.

ottsyd-20170116-3

To be clear, what figure 3 shows is average graduate salaries two years after graduation in real dollars – that is, controlling for inflation.  And what we see here is that in all fields of study, income bops along fairly steadily until 2007 (i.e. class of 2005) at which point things change and incomes start to decline in all six subject areas.  Engineering was down, albeit only by three percent.  But income for business students was down 10%, physical sciences down 16%, and humanities, social sciences and education were down 19%, 20% and 21%, respectively.

This, I shouldn’t need to emphasize, is freaking terrible.  Actual employment rates (link to: previous) may not be down that much but this drop in early graduate earnings is  pretty disastrous for the majority of students.  Until a year or two ago I wasn’t inclined to put a lot of weight on this: average graduate earnings have always popped back after recessions.  This time seems to be different.

Now as I said yesterday, we shouldn’t be too quick to blame this on a huge changes economy to which institutions are not responding; it’s likely that part of the fall in averages comes from allowing more students to access education in the first place.  As university graduates take up an increasing space on the right-hand side of an imaginary bell-curve representing all youth, “average earnings” will naturally decline even if there’s no overall change in the average or distribution of earnings as a whole.  And the story might not be as negative if we were to take a five- or ten-year perspective on earnings.  Ross Finnie has done some excellent work showing that in the long-term nearly all university graduates make a decent return (though, equally, there is evidence that students with weak starts in the labour force have lower long-term earnings as well through a process known as “labour market scarring”).

Whatever the cause, universities (and Arts faculties in particular) have to start addressing this issue honestly.  People know in their gut that university graduates’ futures in general (and Arts graduates in particular) are not as rosy as they used to be. So when the Council of Ontario puts out a media release, as it did last month, patting universities on the back for a job well-done with respect to graduate outcomes, it rings decidedly false.

Universities can acknowledge challenges in graduate without admitting that they are somehow at fault.  What they cannot do is pretend there isn’t a problem, or shirk taking significant steps to improve employment outcomes.

January 16

Ever-bleaker Graduate Employment Data?

So just before I quit blogging in December, the Council of Ontario Universities released its annual survey of graduate outcomes, this time of the class of 2013.  The release contained the usual platitudes: “future is bright”, “vast majority getting well-paying jobs”, etc etc.   And I suppose if one looks at a single year’s results in isolation, one can make that case.  But a look at longer-term trends suggests cause for concern.

These surveys began at the behest of the provincial government seventeen years ago.  Every graduating cohort is surveyed twice: once six months after graduation and once two years after graduation.  Students are asked questions about their employment status, their income and about the level of relationship between their job and their education.  COU publishes only high-level aggregate data, so we don’t know about things like response rates, but the ministry seems pleased enough by data quality, so I assume it’s within industry standards.

Figure 1 shows employment rates of graduates six months and two years out.  At the two-year check point, employment rates fell by four points in the immediate wake of the 2008-9 recession, (be careful in reading the chart: the x-axis is the graduating class, not the year of the survey, so the line turns down in 2006 because that’s the group that was surveyed in 2008).  Since then it has recovered by a little more than a point and a half, though further recovery seems stalled.  At the six-month point, things are much worse.  Though employment rates at this point are no longer falling, they remain stubbornly seven percentage points below where they were pre-recession.

Figure 1: Employment Rates, Ontario University Graduates, 6 Months and 2 Years Out, by Graduating Class, 1996-2013

OTTSYD 20170115-1

If you want to paint a good story here, it’s that employment rates at 2 years out are still within three percentage-points of their all-time peak, which isn’t terrible.  But there doesn’t seem much doubt that students are on average taking a bit longer to “launch” than they used to; employment rates six months out seem to have hit a new, and permanently lower floor.

Now, take a look at what’s happening to starting salaries.  As with the previous graph, I show results for at both the six-month and the two-year mark.

 

 Figure 2: Average salaries, Ontario University Graduates, 6 Months and 2 Years Out, by Graduating Class, 1996-2013, in $2016

OTTSYD 20170115-2

What we see in Figure 2 is the following:  holding inflation constant, during the late 1990s, recent graduates saw their incomes grow at a reasonably rapid clip.  For most of the 2000s, income was pretty steady for graduates two years out (less so six months out).  But since the 2008 recession, incomes have been falling steadily for several years; unlike the situation with employment rates, we have yet to see a floor, let alone a bounceback.  Real average incomes of the class of 2013 six months after graduation were 11% lower than those of the class of 2005 (the last fully pre-recession graduating class); at 2 years out the gap was 13%.  Somehow these points did not make it into the COU release.

That, frankly, is not good.  But it seems to me that we need to hold on a little bit before hitting panic buttons about universities being a bad deal, not being relevant to shifting labour market, etc, etc.  Sure, the drop-off in both employment rates and incomes started around the time of the recession and so it’s easy to create a narrative around changed economy/new normal, etc etc.  But there’s something else that probably playing a role, and that’s an increase in the supply of graduates.

 

Figure 3: Number of Undergraduate Degrees Awarded, Ontario, 1999-2013

OTTSYD 20170115-3

The other big event we need to control for here is the massive expansion of access to higher education.  In 2003, the “double-cohort” arrived on campus and that forced government to expand institutional capacity, which did not subsequently shrink.  Compared to the year 2000, the number of graduates has increased by over 50%; Such an expansion of supply must have had some effect on average outcomes. It’s not simply that there are more students competing for jobs – something one would naturally assume would place downward pressure on wages – but also, the average quality of graduates has probably dropped somewhat.  Where once graduates represented the top 20% of a cohort in terms of academic ability, now they probably represent the top 30% or so.  Assuming one’s marginal product in the labour market is at least loosely tied to academic ability, that would also predict a drop in average post-graduation incomes.  To really get a sense of what if anything has changed in terms of how higher education affects individuals’ fortunes in the labour market, you’d want to measure not average income vs. average income, but 66th percentile of income now vs. 50th percentile of income fifteen years ago.  Over to you, COU, since you could make the microdata public if you wanted to.

In short, don’t let institutions off the hook on this, but recognize that some of this was bound to happen anyway because of access trends.

More graduate income data fun tomorrow.