In the higher education advocacy business, an unhappily large proportion of the research used is of the correlation = causation type.  For instance, many claim that higher education has lots of social benefits like lower crime rates and higher rates of community volunteering on the grounds that outcomes of graduates are better than outcomes of non-graduates in these areas.  But this is shaky.  There are very few studies which look at this carefully enough to eliminate selection bias – that is, that the people who go to higher education were less disposed to crime/more disposed to volunteering to begin with.  The independent “treatment” effect of higher education is much more difficult to discern.

This applies in spades to studying the question of the effects of study abroad.  For instance, one widely quoted study  of the Erasmus program showed that five years after graduation, unemployment rates for graduates who had been in a study-abroad program were 23% lower than for those who did not.  But this is suspect.  First of all “23% lower” actually isn’t all that much for a population where unemployment is about 5% (it means one group has unemployment of 4% and the other 5%, more or less).  Second of all, there is a selection bias here.  The study-abroad and non-study abroad populations are not perfectly identical populations who differ only in that they have been given different “treatments”: they are different populations, one of which has enough drive and courage to pick up sticks to move to another country and (often) study in another language.  It’s quite possible they would have had better employment outcomes anyways.  You can try to limit bias by selecting a control group that is similar to the study abroad population by selecting a group that mimics them in terms of field of study, GPA, etc, but it’s not perfect and very few studies do so anyway (a very honourable mention here to the GLOSSARI project from Georgia headed by Don Rubin)

(Before we go any further: no, I don’t think employability skills are the only reason to encourage study abroad.  I do however think that if universities and colleges are going to frame their claim for more study abroad in economic terms – either by suggesting students will be more employable or making more general claims of increasing economic competitiveness – then it is incumbent on them to actually demonstrate some impact.  Claiming money on an economic imperative and them turning around and saying “on that doesn’t matter because well-rounded citizen” doesn’t really wash.

There are other ways of trying to prove this point about employability, of course.  One is to ask employers if they think study abroad matters.  They’ll usually say yes, but it’s a leap of faith to go from that to saying that study abroad actually is much a help in actually landing a job.  Some studies have asked students themselves if they think their study abroad experience was helpful in getting a job.  The answer is usually yes, but it’s hard to interpret what that means, exactly.

Since it’s difficult to work out directly how well internationalization is helping students get jobs, some people try to look at whether or not students get the skills that employers want (self-discipline, creativity, working in teams, etc).  The problem with this approach of course, is that the only real way to do this is through self-assessment which not everybody accepts as a way to go (but in the absence of actual testing of specific skills, there aren’t a whole lot of other options).  Alternatively, if you use a pre-post evaluation mechanism, you can at least check on the difference in self-assessment of skills over time, which might then be attributed to time spent in study abroad.  If that’s still not enough to convince you (if, for instance, you suspect that all students self-assessments would go up over the space of a few months, because all students are to some degree improving skills all the time), try a pre-post method with a control group, too: if both groups’ self-assessments go up, you can still measure the difference in the rate at which the self-reported skills increase across the two groups.  If they go up more for study-abroad students than for stay-at-homes, then the difference in the rates of growth can, cautiously, be attributed to the study abroad period.

Basically: measuring impacts takes time, and is complicated.  And despite lots of people in Canada avowing how important outbound mobility is, we never seem to take them time, care and expense to do the measurement.  Easier, I suppose, to rely on correlations and hope no one notices.

It’s a shame really because I think there are some interesting and specifically Canadian stories to tell about study abroad.  More on that tomorrow.