HESA

Higher Education Strategy Associates

May 24

Measuring Graduate Quality

A few months ago, I was in a discussion with a number of colleagues about how one should go about measuring how well universities and colleges prepare students for the labour market. It’s a tough question to answer.

Employment rates aren’t helpful because those move with the economic cycle (and in places like Alberta with tight labour markets, low unemployment might be more of a sign of desperation for warm bodies than it is of educational quality). Employer satisfaction surveys are tough to interpret because a) the respondent doesn’t always know a particular employee, and b) these surveys never disaggregate the responses of employers in areas which are related and unrelated to the field of study, which means there is a lot of valuable information being lost. Subject matriculation tests are obviously out.

So what to do?

I don’t think there’s any choice but to rely on input from employers in some form. I mean, at the end of the day, you have to assume they know something about whether their workers are getting the job done or not.

Where I think current employer surveys go wrong is in trying to establish what employers think of each individual graduate so as to rate the outputs of individual college or university programs. This, frankly, is nuts. Major curricular differences between programs at different institutions these days aren’t all that common. Differences between individuals are as likely to be due to innate qualities and personal characteristics as they are to the instruction received.

Anyways, it occurred to me that surely what matters most from a public policy perspective is whether the system as a whole is producing graduates that meet the needs of employers. So why not ask employers that question instead of the whole present rigamarole?

But – and this is crucial – let’s not ask whether employers are “satisfied” with the skills new graduates bring to the workplace. That’s actually a very hard question to interpret (are they saying “satisfied” because they are polite and Canadian? Are they saying “dissatisfied” because they have inflated expectations of the amount of job-specific skills graduates should have?). Instead, let’s ask them, “Are the recent college/university graduates you hired this year better or worse than the ones you hired five years ago?”

This is a nice simple question that gets to the heart of the issue. You’d know right away whether engineering firms thought that engineering education as a whole was keeping up with the times or not. Or child care centres, or accounting firms or the public service.

That’s what matters. That’s the major reason we fund education so heavily. Yet it’s not a question we regularly pose. Time for a change.

May 23

Discontent over Employment

It’s that time of year when the subjects of “education” and “jobs” are inextricably linked. When the economy is good, the stories are about go-getting recent graduates, and how they’re changing the world, etc. Universities then use this publicity to argue for more research money. “Look at all the great stuff our graduates do,” they say. “Our education gets kids jobs!”

When the economy is bad, on the other hand, universities are subject to variations of Margaret Wente’s recent irritating rant. “You’re failing our kids,” they say. “Why are students spending tens of thousands of dollars on sociology degrees that don’t lead to jobs? Why can’t universities be more job-oriented?” To which the universities reply: “Why are you picking on us? We didn’t start the recession!”

This dance plays itself out with metronomic regularity. Read newspapers from 1982 or 1993 and they all sound like Ms. Wente today. Read them in the midst of a two-decade boom and apparently universities are so fantastic that we can’t shovel money into them fast enough.

The basic problem here is that neither side is being entirely honest.

Start with the university position. Universities know perfectly well that in many fields of study they have no idea how well their education “gets kids jobs.” To a significant extent the value of a degree comes not from the value of any skills obtained (“human capital theory”) but simply from having been admitted to university and demonstrating an ability to stick it out through four years of instruction (“signaling theory”). The problem is that universities have to assert that human capital theory is true, because that’s almost the entire rationale for the large sums of public funding they receive. So when times are good, they’re incapable of restraining themselves from proclaiming their “successes” in this area – which unfortunately makes it hard to escape the “failure” tag when times are bad.

Then there’s the pro-Wente position. If they were being honest, these critics would recognize that exactly the same education that they claim is completely useless was delivering graduate unemployment rates of 3-4% just a couple of years ago. Has the world of work changed so massively? Has tuition changed massively, or student debt? No, we just happen to be going through the worst financial crisis since the 1930s. The independent variable here is Lehman Brothers, not sociology curricula.

That’s not to say there aren’t good reasons to think more deeply about curriculum and how it can be made more valuable to students. But it is to say that this debate involves a disappointingly high ratio of smack-talk to sense. Everybody needs to take a valium and focus on real issues.

May 22

Does Debt Affect Career Choice?

A lot of hypotheses about the negative effects of student debt (some of which I was responsible for, 15 or so years ago), have, over time, been shown to be wrong. The one about debt being a serious deterrent to access, for instance (at least at current levels of borrowing); or the one about how increased student debt delays family formation. But what about the hypothesis that higher levels of student debt might leading students to take “jobs that pay a lot of money” rather than “jobs that are socially useful”?

One paper which seemed to confirm this hypothesis was Constrained After College: Student Loans and Early Career Occupational Choices by Jesse Rothstein and Cecilia Rouse. Using administrative, financial and alumni data from an “anonymous university” (blatantly obviously Princeton), the authors examined the effects on employment of an institutional aid reform which saw loans replaced entirely with grants. They found that after the policy switch, students who received aid took jobs with lower average incomes, while students who were not on aid saw no change in their income patterns. Most of the action happened at the bottom end of the scale – there was no reduction in the proportion going on to investment banking and consulting – the shift was mainly from lower-paying private sector jobs into the non-profit sector.

This is an excellent paper, but still – it’s Princeton. Can you really generalize from that kind of sample?

We thought we’d attack the question from another angle by asking students what they will be looking for in a job after graduation: earning potential, balance between work and personal life, fulfilling work, mentorship and learning possibilities, work involves creative rather than routine tasks, flexible working schedule. Then we paired the different options against each other and asked students to tell us which they preferred (matched pairs are a great way to reveal preferences). Finally we looked at how students answered the question based on whether or not they had loans. If the debt hypothesis is true, then you’d expect loan recipients to sacrifice all that good stuff for a job with a higher earnings potential

Here’s how it turned out:

Interesting, huh? And yeah, we checked – the result doesn’t change the closer students get to graduation (and accumulate more debt).

Now maybe the real effect happens in the few months after graduation when they’re suddenly confronted with the reality of paying back loans (though CSL’s various income-contingent features should mitigate that at least somewhat). But as they head out into the workforce there’s little evidence that a preoccupation with earnings is, at present at least, a major outcome of student indebtedness.

May 18

Size Matters

Did this make it through your spam filters? Here’s hoping.

Ok, so everyone knows from even a casual glance at student satisfaction data that there’s a correlation between institutional size and satisfaction. Students, on average, prefer small schools to big schools. Makes them feel more at home. Less of a leap from high school. The accompanying small class sizes don’t hurt either (the relationship between size and satisfaction is of course one reason why the Globe and Mail’s Canadian University Report takes care to separate institutions by size before comparing them).

As the graph below shows, on a nine-point scale, a university tends to lose about a tenth of a point for every extra 6,000 students they enrol. That sounds small, but when you consider that all but two institutions in Canada receive a score of between six and eight, even small differences can have a big effect on where schools place relative to one another.

Relationship between School Satisfaction and Institutional Enrolment

So, what would happen if you normalized satisfaction scores for size? That is to say, what if, instead of measuring an absolute value of satisfaction, we measured the distance from that nice little diagonal trend line?

The answer is that not much would change for some institutions – especially those that already have enrolments in the 10-15,000 range – but there would be some very big adjustments for some institutions at the top and the bottom.

At one end of the scale, there are a number of institutions whose poor performance in satisfaction rankings can be ascribed mainly (but not entirely) to the size of the institution – they actually do as well as (or maybe even better than) expected, given their size. With size-normalized scores, the University of Toronto would jump 23 places in the table out of 62 institutions. UBC would rise 19 places, the University of Alberta 17, McGill 16, Concordia 15 and Ryerson 11.

On the other end of the scale it’s a different story: there are a number of institutions whose position at or near the top of the table are due to a large degree to their size and their size alone. Normalized for enrolments, UPEI would drop 14 places, Cape Breton, Brandon and UNBC would drop 11 places, while Thompson Rivers and Nipissing would drop 10 places each.

You can make an argument either way for whether normalizing for size is the right way to display the data. Clearly, students like smallness and that needs to be reflected somehow. But then again, you also want to be able to display data in a way that rates institutions base on what they do, not on their size. I’d be interested to hear from anyone with ideas on how to do both in the next CUR.

May 17

The New York Times Swings and Misses

Sure signs of spring: baseball is back (and so is Vlad!), Ottawa is full of tulips, Quebec students are demonstrating in the buff and newspaper editors are turning their attention to student debt.

Exhibit A: the Globe’s spread on debt last Saturday (full disclosure: HESA supplied some of the data the Globe published).

Exhibit B: the cover of Sunday’s New York Times – “A Generation Hobbled by the Soaring Cost of College.”

Now, geeky wonks that we are, our first reaction to the appearance of higher education policy issues front and centre in the press gets us excited all out of proportion. And, nine times out of ten, our excitement turns to righteous indignation as soon as we read what’s written. For some inexplicable reason, journalists don’t tend to report stories the way we would want. So we eat breakfast. Outrage fades. We go on with our weekend.

But, as my insightful colleague Don Heller points out, there’s some crucial sleight of hand going on in that Times piece (and a major factual error) that, given the Times’s outsized influence, could seriously degrade the quality of public debate around higher education affordability. The Times chose to illustrate its story – which reported the average U.S. Bachelor’s-degree-holder debt (excluding those who have no debt) at $23,300 in 2011 – with the story of a hard-luck graduate of a private university marketing program with no prospects and $120,000 in debt.

Don points out the sequence of lousy decisions that have culminated in this student’s moving back in with her parents, and pinpoints how anomalous her story is. Moreover, it’s become clear that the Times misread survey data to conclude that 94% of graduates accumulate debt, when the actual figure is 62%. (The Times’s corrected the error on Wednesday.)

To its credit, the Globe chose to illustrate its piece about average student debt with a story about a graduate with… average student debt.

As we pointed out a few weeks ago, deriving meaning from the stats on student debt isn’t as obvious as it may seem. Newspaper stories that namecheck means and medians but focus on the 1% of graduates in dire straits ($120,000 in debt! “But when I graduate, I’m going to owe like $900 a month. No one told me that”!) royally undermine efforts to have a reasonable debate about access to education and student debt. By presenting the extreme as the norm, newspapers may get more hits, but they betray their public service mission.

We can surely expect better from the New York Times.

May 16

Learning From Other Policy Areas: Housing

Sometimes, I think we spend too much time trying to learn from policy in other countries and not enough time learning from related policy areas in our own country.

Take housing, for instance – probably higher education’s closest relative in policy terms. Choosing a home, like choosing an undergraduate degree, is a major decision with enormous financial implications, but both can be foreseen many years in advance, allowing people to plan for the purchase. Shelter and education are both considered to be “rights,” but we ask people to pay for them both. In addition to their status as public goods, a substantial benefit of both housing and higher education are private. The subsidy picture is a similar mix of non-repayable aid for the poor (grants in our field, social housing and housing vouchers in the other), and assistance via loans for everyone else (CSLP, CMHC insurance). Finally, both are to some degree positional goods; not every degree is from Harvard, not every house is in Westmount, and the location of your alma mater/house determines its value to a substantial degree.

So, what can we learn about ourselves from looking at housing?

First, education policy is big on encouraging savings – so why is there no Canada Education Savings Grant equivalent for housing? (Ok, arguably the home mortgage interest tax deduction in the U.S. is pretty close.) But in Canada, I’d guess the reason we don’t is that everyone would think it’s a stupid idea to help those most able to help themselves. So why doesn’t that logic apply to education too? Hmmm.

Second, housing policy also has a much more limited view about the state’s role in minimum provision. There is a “right to housing,” but that tends to mean things like emergency shelter, a spot in sometimes quite decrepit social housing, etc (that is to say, a good which is significantly inferior in quality to what most people have). In higher education, on the other hand, that right is seen as a right to a shot at just about any form of education, provided you think you can pay back the loan.

Third, there is the way that loan assistance is targeted. Student loans are needs-tested, but CMHC mortgage insurance isn’t. Why is that? Should there be a needs test on loans? Should we introduce one on mortgage insurance? Why would we do one and not the other?

Obviously, there are some differences between the two sectors which make some policy differences natural. The most is the nature of the underlying asset; a lender can repossess a house but can’t repossess a degree. But the similarities are enough that wonks in both fields could benefit from spending more time learning from one another.

May 15

Ranking Higher Education Systems

As you may know, a new ranking of national systems of higher education was released a couple of weeks ago. The ranking was put together by a team of Monash University professors led by Ross Williams, and published by the Melbourne Institute and Universitas 21, a global alliance of 24 research intensive universities. It’s not the first attempt to rank national systems – we here at HESA have done two iterations of a global ranking on affordability and accessibility – but it is probably the most ambitious.

The ranking covers twenty indicators over four themes – Resources, Environments, Connectivity and Outputs. For those keeping score, Canada came third overall behind the U.S. and Sweden. We were top in the resource category (there goes the underfunding argument!), third in the output category and mid-table in both environment and connectivity. Not bad, huh?

But you need to take these results with a big grain of salt for a couple of reasons. The first is that the “environment” indictors are mostly qualitative and there’s no published source available to check the data on two of them. The scoring here is simply bizarre, though. I know there’s no such thing as a validity check in ranking, but surely to God the fall-down-laughing test should have given the authors pause before ranking Canada behind Ukraine on this one.

Similarly, the choice of measures and data sources for outputs tends to benefit the U.S. disproportionately. Awarding points for the percentage of the population over 24 with a degree privileges those countries that massified a long time ago (i.e., the U.S.). Using crude gross enrolment ratios as a measure of access privileges countries where students stay enrolled for a long time (there’s also a nerdy comparability problems on the numerator of this statistics in that the U.S. counts its community college students and we don’t). And using researchers per head of population and unemployment rates as measures of university outputs when in fact institutions have at best partial control over these numbers is just silly.

One thing that is disappointing about these rankings is that in many ways, they just measure how close national systems are to a Northern European ideal. If you’re a rich country that drank the Humboldt kool-aid a long time ago, you tend to do well. If you’re poor and have largely teaching-oriented universities, you don’t. I suppose since the ideal of “world-class” research universities has become a global standard of sorts, this makes some sense. But a measure which was more sensitive to levels of national income would have been nice.

Overall: it’s a good first attempt, but there’s still some room for improvement.

May 14

New Possibilities in African Higher Education

As I’m working in Ghana this month, I thought I’d share a few stories about higher education here in Africa. What’s occupying my thoughts these days is the educational production function (yes, really).

The most amazing thing “western” education systems ever did was to get students to learn on their own, thus reducing both the number of professors required to teach a given number of students and, as a result, the cost of education. Teaching students to “learn how to learn” thus isn’t just cant – it’s the heart of the most important productivity revolution in world education.

But in Africa, libraries are virtually non-existent and until recently electronic resources were inaccessible. Since the tools to “learn to learn” didn’t exist, the only way “learning” can occur is in the classroom. Hence the need for African students to spend 25 to 30 hours in the classroom each week, which is why African professors teach more than ours do.

Now, though, there’s the potential of adopting the western education production function. The libraries are still empty, of course, but the ICT revolution has put an incredible amount of information and resources at the fingertips of anyone with an iPad or laptop, and the relentless fall in computing costs means an increasing number of students are able to access that technology. Which means – in theory at least – they could move to a cheaper, less labour-intensive and generally more “western” university system any time now. That would be a major development, as it makes it possible to expand the higher education system at much lower cost. In a continent with a burgeoning youth population and soaring demand for education, such a game-changer is desperately needed.

The problem is that the new regime reduces the need for professors in the short term (in the longer-term, the effect is probably neutral as demand for education rises). Predictably, therefore, there aren’t a lot of institutions stepping forward to use these new technologies to change the way they deliver education. The mostly church-run private university sector won’t take up the opportunity because, being prestige-seekers like everyone else, they are mainly concerned with trying to ape whatever the most venerable university in the region is doing. And African governments won’t push the idea because reducing employment – even temporarily – causes unrest, and the Arab Spring has made most governments very risk-averse.

The likely best option to catalyze this education revolution is a foreign provider, unencumbered by existing contracts and bringing the prestige of being “western” to come in and establish this model, and hope it spreads by example. There is enormous opportunity to do good here – but will anyone step forward to make it happen?

May 11

Hooked on School

What do Canadian students do when they’ve finished their university studies? And how do they differ from students in other parts of the world? We recently had the opportunity to examine country-level graduate surveys around the world.

Now, there are important caveats – no two countries conduct the same survey among the same exact population of graduates at the exact same time (and international data agencies like the OECD restrict most of their graduate analysis to fairly basic indicators, such as employment rates and earnings). Fortunately, centrally-coordinated surveys in Canada, Australia, France, Germany, New Zealand, Sweden, the United States and the United Kingdom permit meaningful international comparisons of life after a Bachelor’s degree.

So how do Canadian Bachelor’s degree-holders compare? For one thing, those who decide to work (excluding those who work and study part-time) report the second-highest earnings among the countries examined here, along with the U.S. – around $45,000 annually.* German students with a “traditional degree” report earnings of CAD $55,000.

More interesting, though, is the proportion of Canadian graduates who pursue further education. According to Statistics Canada’s National Graduates Survey, 42% of Bachelor’s degree-holders from the class of 2005 had pursued a new course of study by 2007; only 62% of Canadians were working (some did both). No one, other than German graduates of “new” degree programs, studies as much after completing an undergraduate degree than Canadians – even those who were surveyed in other countries much longer after graduation were less likely to return to school.

Of the 42% of Canadian graduates who went on to pursue an additional program (the number of students who pursued any kind of schooling, either within the context of a program or one or more individual courses, is closer to 75%), only 16% had completed their studies when surveyed two years after graduation.

Of course, not all students are as gung-ho about pursuing post-graduate education. Looking at graduates at all levels, while 62% of life-science graduates and 56% of humanities graduates went on to pursue another program of study, the same was true for less than one-third of architecture and engineering students. Education graduates were most likely (64%) to pursue individual courses after graduation; however fewer than 20% undertook a structured program.

Whether a Bachelor’s degree doesn’t confer the benefits Canadians students anticipate or whether it cultivates a thirst for learning that can only be quenched in the classroom isn’t clear. What is certain, though, is that most Canadian undergraduates aren’t ready to stop studying.

* Australian graduates actually reported higher earnings – $61,000 vs. $45,000 in Canada and the U.S., but they were surveyed five years after graduation, compared to one to two years in Canada and the States.

May 10

What We’re Reading Now: Creating the Market University

If you’re interested in reading about the major events that shaped the evolution of universities in the twentieth century, then you could do a lot worse than invest a few hours in reading Elizabeth Popp Berman’s Creating the Market University: How Academic Science Became an Economic Engine.

Most histories of science rightly point to Vannevar Bush’s 1945 essay “Science: the Endless Frontier” as the point at which the U.S. government definitively committed itself to funding university science in a big way. The logic was that academic science, which acted as a reservoir of ideas on which industry could draw, was a public good and needed to be supported as such. Berman argues that from the mid-1970s to the mid-1980s, the science-as-reservoir was replaced by a science-as-economic-engine logic that pushed American scientists to look at how the market valued their research and to seek out opportunities to work with outside partners to increase the value of their knowledge.

What distinguishes Berman from others looking at similar phenomena is her refusal to simply label this as an exercise in neo-liberalism. To be sure, she grants that some of the developments that pushed universities in this direction – notably the 1978 tax changes that favoured venture capital, the Chakrabarty Supreme Court decision – were rooted in free-market logic. But many others of them were rooted in the thinking of economists like Kenneth Arrow, who pointed out that markets on their own could produce suboptimal investments in R&D.

In fact, she points out the real shift in thinking happened not in universities but in governments. Dismayed by U.S. economic performance in the 1970s, state and federal governments began to focus on innovation as a key driver of economic performance. This led them to incentivize institutions to get their knowledge into the hands of American industry so as to make better products and produce more jobs. Far from the shift to the market being a result of government cutbacks (cuts, for the most part, happened in physics and chemistry, whereas the push to market was led by academics in biology and engineering), it was in fact the result of a shift in how governments thought about economic growth.

It’s a very American history, of course, but Canada went through a very similar process ten years later in the 1980s and 1990s, and most of the rest of the world followed suit as well. Indeed, thirty-five years on from the events Berman recounts, good market-focussed academic science is now the gold standard for universities everywhere. Well-written and admirably succinct, this tale of how the modern university acquired these market ideals is definitely worth a read.

Page 22 of 40« First...10...2021222324...3040...Last »