HESA

Higher Education Strategy Associates

Tag Archives: Survey Methodology

September 15

Why our Science Minister is Going to be Disappointed in Statscan

Last week Statscan sent me a consultation form asking my opinions about their ideas on how to change UCASS (the University and College Academic Staff Survey, which like most Statscan products containing the word “college” does not actually include the institutions most of us call “colleges” i.e. community colleges).  I’ve already said something about this effort back here to the effect that focussing so much effort on data collection re: part-time staff is a waste of time, but the consultation guide makes me think Statscan is heading into serious trouble with this survey reboot for a completely different set of reasons.

Remember that when the money for all this was announced, the announcement was made by our Minister of Science, Kristy Duncan.  One of her priorities as Minister has been to improve equity outcomes in scientific hiring, particularly when it comes to things like Canada Research Chairs (see here for instance).  The focus of her efforts has usually been gender, but she’s also interested in other equity populations – in particular, visible minorities, Indigenous peoples, and persons with disabilities.  So one of the things she charged Statscan with doing in this revived UCASS (recall that Statscan cut the program for five years as a result of Harper-era cuts) is to help shine a light on equity issues in terms of salaries, full-time/part-time status, and career progression.

This is all fine except for one tiny thing.  UCASS is an not a questionnaire-based instrument.  It’s an administrative survey.  That means institutions fill in a complicated set of sheets to provide Statscan with hundreds of different aggregated data cuts about their institution (what is the average salary of professors in Classics?  How many professors in chemical engineering are female?  Etc).  In order to use UCASS to address the demographic questions Duncan wants answered, institutions would first need to know the answer themselves.  That is, they would need to know precisely which instructors have disabilities, or which are “visible minorities”, just as they currently know everyone’s gender.  Which means they would need to find a way to make such disclosures mandatory, otherwise they would not be able to report to Statistics Canada.

I tried this idea out on my twitter audience over the weekend.  Let’s just say it did not go over very well.  A significant number of responses were, essentially: “over my dead body do I give this information to my employer.  If Statscan wants to know this, they can ask me directly.”

Well, yes, they could I suppose, but then the resulting data couldn’t be linked to administrative information on rank and salary without getting each faculty member’s permission, which I can see not always being forthcoming.  In addition, you’d have all sorts of non-response bias issues to deal with, especially if they tried to do this survey every year – my guess is most profs would simply ignore the survey after year 2.  And yes, you’d have to do it frequently because not all disabilities are permanent.

Here’s my suggestion.  Statscan should actually do two surveys.  Keep UCASS more or less the way it is, extend it to colleges (some of whom will take a decade to report properly but that’s life) and part-timers (if they must – frankly, I think more people would be interested in data on non-academic staff than in data on part-time staff) but don’t mess around with the basic structure or try to force professors into reporting on their demographic characteristics – other than gender, which is already in there – to their employers because that’s just more trouble than it’s worth.  Then, every five years or so, do a second survey so in which you take a demographic snapshot of the professoriate as a whole.  It will have mediocre completion rates, but it’s better than nothing.

(In fact, universities and colleges could do this themselves if they wanted to at a cost much lower than whatever Statscan will end up paying, but since they almost never collaborate on creating public data without a gun to their heads it seems like some federal intervention is inevitable if anyone wants this done).

This is not what Minister Duncan asked for, I know.  But it’s the only way to get her the data she wants without causing mayhem on campuses.  Hopefully, pragmatism will prevail here.

September 08

Data on Sexual Harassment & Sexual Assault in Higher Ed-an Australian Experiment

Earlier this year, I raged a bit at a project that the Ontario government had launched: namely, an attempt to survey every single student in Ontario about sexual assault in a way that – it seemed to me – likely to be (mis)used for constructing a league table on which institutions had the highest rates of sexual assault.  While getting more information about sexual assault seemed like a good idea, the possibility of a league table – based as it would be on a voluntary survey with pretty tiny likely response rates – was a terrible idea which I suggested needed to be re-thought.

Well, surprise!  Turns out Australian universities actually did this on their own initiative last year.  They asked the Australian Human Rights Commission (AHRC) to conduct a survey almost exactly along the lines I said was a terrible idea. And the results are…interesting.

To be precise: the AHRC took a fairly large sample (a shade over 300,000) of university students – not a complete census the way Ontario is considering – and sent them a well-thought-out survey (the report is here).  The response rate was 9.7%, and the report authors quite diligently and prominently noted the issues with data of this kind, which is the same as bedevils nearly all student survey research, including things like the National Survey of Student Engagement, the annual Canadian Undergraduate Research Consortium studies etc etc.

The report went on to outline a large number of extremely interesting and valuable findings.  Even if you take the view that these kinds of surveys are likely to overstate the prevalence of sexual assault and harassment because of response bias, the data about things like the perpetrators of assault/harassment, the settings in which it occurs, report of such events and the support sought afterwards are still likely to be accurate, and the report makes an incredible contribution by reporting these in detail (see synopses of the reports  from CNN, and Nature).  And, correctly, the report does not reveal data by institution.

So everything’s good?  Well, not quite.  Though the AHRC did not publish the data, the fact that it possessed data which could be analysed by institution set up a dynamic where if the data wasn’t released, there would be accusations of cover-up, suppression, etc.  So, the universities themselves – separate from the AHRC report – decided to voluntarily release their own data on sexual assaults.

Now I don’t think I’ve ever heard of institutions voluntarily releasing data on themselves which a) allowed direct comparisons between institutions b) on such a sensitive subject and c) where the data quality was so suspect.  But they did it.  And sure enough, news agencies such as ABC (the Australian one) and News Corp immediately turned this crap data into a ranking, which means that for years to come, the University of New England (it’s in small-town New South Wales) will be known as the sexual assault capital of Australian higher education.  Is that label justified?  Who knows?  The data quality makes it impossible to tell.   But UNE will have to live with it until the next time universities do a survey.

To be fair, on the whole the media reaction to the survey was not overly sensationalist.  For the most part, it focussed on the major cross-campus findings and not on institutional comparisons.  Which is good, and suggests that some of my concerns from last year may have been overblown (though I’m not entirely convinced our media will be as responsible as Australia’s).  That said, for data accuracy, use of a much smaller sample with incentives to produce a much higher response rate would still produce a much result with much better data quality than what the ARHC did, let alone the nonsensical census idea Ontario is considering.  The subject is too important to let bad data quality cloud the issue.

 

Erratum: There was a data transcription error in yesterday’s piece on tuition.  Average tuition in Alberta is $5749 not $5479, meaning it is slightly more expensive than neighbouring British Columbia, not slightly less.

April 05

Student/Graduate Survey Data

This is my last thought on data for awhile, I promise.  But I want to talk a little bit today about what we’re doing wrong with the increasing misuse of student and graduate surveys.

Back about 15 years ago, the relevant technology for email surveys became sufficiently cheap and ubiquitous that everyone started using them.  I mean, everyone.  So what has happened over the last decade and a half has been a proliferation of surveys and with it – surprise, surprise – a steady decline in survey response rates.  We know that these low-participation surveys (nearly all are below 50%, and most are below 35%) are reliable, in the sense that they give us similar results year after year.  But we have no idea whether they are accurate, because we have no way of dealing with response bias.

Now, every once in awhile you get someone with the cockamamie idea that the way to deal with low response rates is to expand the sample.  Remember how we all laughed at Tony Clement when he claimed  the (voluntary) National Household Survey would be better than the (mandatory) Long-Form Census because the sample size would be larger?  Fun times.  But this is effectively what governments do when they decide – as the Ontario government did in the case of its sexual assault survey  – to carry out what amounts to a (voluntary) student census.

So we have a problem: even as we want to make policy on a more data-informed basis, we face the problem that the quality of student data is decreasing (this also goes for graduate surveys, but I’ll come back to those in a second).  Fortunately, there is an answer to this problem: interview fewer students, but pay them.

What every institution should do – and frankly what every government should do as well – is create a balanced, stratified panel of about 1000 students.   And it should pay them maybe $10/survey to complete surveys throughout the year.  That way, you’d have good response rates from a panel that actually represented the student body well, as opposed to the crapshoot which currently reigns.  Want accurate data on student satisfaction, library/IT usage, incidence of sexual assault/harassment?  This is the way to do it.  And you’d also be doing the rest of your student body a favour by not spamming them with questionnaires they don’t want.

(Costly?  Yes.  Good data ain’t free.  Institutions that care about good data will suck it up).

It’s a slightly different story for graduate surveys.  Here, you also have a problem of response rates, but with the caveat that at least as far as employment and income data is concerned, we aren’t going to have that problem for much longer.  You may be aware of Ross Finnie’s work  linking student data to tax data to work out long-term income paths.  An increasing number of institutions are now doing this, as indeed is Statistic Canada for future versions of its National Graduate Survey (I give Statscan hell, deservedly, but for this they deserve kudos).

So now that we’re going to have excellent, up-to-date data about employment and income data we can re-orient our whole approach to graduate surveys.  We can move away from attempted censuses with a couple of not totally convincing questions about employment and re-shape them into what they should be: much more qualitative explorations of graduate pathways.  Give me a stratified sample of 2000 graduates explaining in detail how they went from being a student to having a career (or not) three years later rather than asking 50,000 students a closed-ended question about whether their job is “related” to their education every day of the week.  The latter is a boring box-checking exercise: the former offers the potential for real understanding and improvement.

(And yeah, again: pay your survey respondents for their time.  The American Department of Education does it on their surveys and they get great data.)

Bottom line: We need to get serious about ending the Tony Clement-icization of student/graduate data. That means getting serious about constructing better samples, incentivizing participation, and asking better questions (particularly of graduates).  And there’s no time like the present. If anyone wants to get serious about this discussion, let me know: I’d be overjoyed to help.

February 23

Garbage Data on Sexual Assaults

I am going to do something today which I expect will not put me in good stead with one of my biggest clients.  But the Government of Ontario is considering something unwise and I feel it best to speak up.

As many of you know, the current Liberal government is very concerned about sexual harassment and sexual assault on campus, and has devoted no small amount of time and political capital to getting institutions to adopt new rules and regulations around said issues.  One can doubt the likely effectiveness of such policies, but not the sincerity of the motive behind them.

One of the tools the Government of Ontario wishes to use in this fight is more public disclosure about sexual assault.  I imagine they have been influenced with how the US federal government collects and publishes statistics on campus crime, including statistics on sexual assaults.  If you want to hold institutions accountable for making campuses safer, you want to be able to measure incidents and show change over time, right?

Well, sort of.  This is tricky stuff.

Let’s assume you had perfect data on sexual assaults by campus.  What would that show?  It would depend in part on the definitions used.  Are we counting sexual assaults/harassment which occur on campus?  Or are we talking about sexual assaults/harassment experiences by students?  Those are two completely different figures.  If the purpose of these figures is accountability and giving prospective students the “right to know” (personal safety is after all a significant concern for prospective students), how useful is that first number?  To what extent does it make sense for institutions to be held accountable for things which do not occur on their property?

And that’s assuming perfect data, which really doesn’t exist.  The problems multiply exponentially when you decided to rely on sub-standard data.  And according to a recent Request for Proposals placed on the government tenders website MERX, the Government of Ontario is planning to rely on some truly awful data for its future work on this file.

Here’s the scoop: the Ministry of Advanced Education and Skills Development is planning to do two surveys: one in 2018 and one in 2024.  They plan on getting contact lists of emails of every single student in the system – at all 20 public universities, 24 colleges and 417 private institutions and handing them over to a contractor so they can do a survey. (This is insane from a privacy perspective – the much safer way to do this is to get institutions to send out an email to students with a link to a survey so the contractor never sees the names without students’ consent).  Then they are going to send out an email to all those students – close to 700,000 in total – offering $5/per head to answer a survey.

Its not clear what Ontario plans to do with this data.  But the fact that they are insistent that *every* student at *every* institution be sent the survey suggests to me that they want the option to be able to analyze and perhaps publish the data from this anonymous voluntary survey on a campus by campus basis.

Yes, really.

Now, one might argue: so what?  Pretty much every student survey works this way.  You send out a message to as many students as you can, offer an inducement and hope for the best in terms of response rate.  Absent institutional follow-up emails, this approach probably gets you a response rate between 10 and 15% (a $5 incentive won’t move that many students)  Serious methodologists grind their teeth over those kinds of low numbers, but increasingly this is the way of the world.  Phone polls don’t get much better than this.  The surveys we used to do for the Globe and Mail’s Canadian University Report were in that range.  The Canadian University Survey Consortium does a bit better than that because of multiple follow-ups and strong institutional engagement.  But hell, even StatsCan is down to a 50% response rate on the National Graduates Survey.

Is there non-response bias?  Sure.  And we have no idea what it is.  No one’s ever checked.  But these surveys are super-reliable even if they’re not completely valid.  Year after year we see stable patterns of responses, and there’s no reason to suspect that the non-response bias is different across institutions.  So if we see differences in satisfaction of ten or fifteen percent from one institution from another, most of us in the field are content to accept that finding.

So why is the Ministry’s approach so crazy when it’s just using the same one as every one else?  First of all, the stakes are completely different.  It’s one thing to be named an institution with low levels of student satisfaction.  It’s something completely different to be called the sexual assault capital of Ontario.  So accuracy matters a lot more.

Second, the differences between institutions are likely to be tiny.  We have no reason to believe a priori that rates differ much by institutions.  Therefore small biases in response patterns might alter the league table (and let’s be honest, even if Ontario doesn’t publish this as a league table, it will take the Star and the Globe about 30 seconds to turn it into one).  But we have no idea what the response biases might be and the government’s methodology makes no attempt to work that out.

Might people who have been assaulted be more likely to answer than those who did not?  If so, you’re going to get inflated numbers.  Might people have reasons to distort the results?  Might a Men’s Rights group encourage all its members to indicate they’d been assaulted to show that assault isn’t really a women’s issue?  With low response rates, it wouldn’t take many respondents to get that tactic to work.

The Government is never going to get accurate overall response rates from this approach.  They might, after repeated tries, start to see patterns in the data: sexual assault is more prevalent in institutions in large communities than in small ones, maybe; or it might happen more often to students in certain fields of study than others.  That might be valuable.  But if the first time the data is published all that makes the papers is a rank order of places where students are assaulted, we will have absolutely no way to contextualize the data, no way to assess its reliability or validity.

At best, if the data is reported system-wide, the data will be weak.  A better alternative would be to go with a smaller random sample and better incentives so as to obtain higher  response rates.  But if it remains a voluntary survey *and* there is some intention to publish on a campus-by campus basis, then it will be garbage.  And garbage data is a terrible way to support good policy objectives.

Someone – preferably with a better understanding of survey methodology – needs to put a stop to this idea.  Now.

February 29

The Heinous Difficulty in Understanding What Works

The empirical consensus on the question of barriers to access in Canadian education is pretty clear: and among those few secondary school graduates who don’t go on to post-secondary education, affordability is very much a secondary issue (not non-existent, but secondary). The primary issue is that most of these young people don’t feel very motivated by the idea of spending more years in a classroom. It’s a vicious circle: these students don’t identify with education, so they don’t work at it, so they receive poor grades and become even more demotivated.

The problem is that it is easier to identify problems than solutions. Big national datasets like the Youth in Transition Survey (YITS) can help identify relationships between inputs and outputs factors, but are useless at examining the effects of interventions because they simply don’t capture the necessary information. What is needed is more small-scale experimentation with various types of interventions, along with rigorously-designed research to help understand their impacts.

This, by the way, is the tragedy of Pathways to Education. It ought to work because it ticks nearly all the boxes that the literature suggests should ameliorate access. But for some reason there has yet to be any serious attempt to evaluate its outcomes (my bet is that Pathways board members prefer anecdotes to data for fundraising purposes – and given their fundraising success to date, it’s hard to blame them). That’s a shame, because if they are on to something it would be useful to know what it is so that it can be replicated.

Now, one shouldn’t pretend that these evaluations are easy. In the United States, a top-notch research company’s multi-year, multi-million-dollar evaluation of the Upward Bound program is currently the subject of intense controversy because of a dispute regarding how data from different intervention sites was weighted. Do it one way (as the evaluators did) and there’s no significant result, do it another and a significant effect appears.

The Upward Bound controversy is a shame because of its likely chilling effect on research in this area. Governments might well question the point of funding research if the results are so inconclusive. But the nature of social interventions is such that there are hundreds of factors that can affect outcomes and hence research is always going to be somewhat tentative.

So what’s the way forward? Research can’t be abandoned, but probably needs to go small-scale. Having lots of small experimental results aggregated through meta-analysis will in the end probably yield far better results than will mega-experiments or more large-scale surveys. It might take a little longer, but it’s both more financially feasible and more likely to deliver durable results.