Universities (and to a lesser extent colleges) are sometimes accused of being change-resistant.  Various stakeholders have lots of valuable feedback to give, so the critique goes, but institutions Just. Don’t Listen.   This critique has some merit but misses the mark in some major ways.  Institutions solicit and receive feedback all the time.  I just don’t think the questions being asked are always very good ones, and the people whose opinions are being solicited are not always the right ones.

Here’s an example: we spend way too much time asking students about satisfaction (either with the institution as a whole, or with specific services or courses), and not nearly enough time asking students about their actual experiences on- and off-campus.  The National Survey of Student Engagement (NSSE) showed everybody the way in this respect, but most schools never extended that approach to thinking about services or individual classes/professors.  The result is that too much of the information collected is shallow and doesn’t allow for meaningful improvements.

Similarly: we ask a lot of questions at the institutional level (surveys like NSSE and the various Canadian Undergraduate Survey Consortium instruments), and at the level of the individual course (various types of teaching evaluation).  But the academic experience is fundamentally built around disciplines and programs, and we ask remarkably few questions about them.  Why?  Because, fundamentally, accountability relationships within higher education have been focussed on individuals and institutions and almost nothing in between.  This is, to put it mildly, bananas.  We could probably get much more interesting data about the quality of teaching if we divorced it from individual “accountability” exercises and focussed it instead on program-level improvement.  So why not change?

And again: we spend far too much time talking to students and not nearly enough talking to alumni.  Some student feedback is important.  But if the reason for gathering feedback is to improve academic programs, why do we spend next to no time talking to graduates and ask program-specific questions.  We almost never ask graduates about their career trajectories, or how they use specific skills/knowledge/competency over the course of their careers.  We almost never ask about which courses were least helpful or what skills would, in retrospect, they realised they were missing.  That is to say, we don’t ask alumni about anything that matter for program improvement.  Why?

In the same vein: why is it that its really only faculties who have to deal with accreditation that talk to employers on a regular basis?  Most Arts and Science faculties have almost no idea who the largest employers of their graduates are, let alone have regular discussions with them about what they like and don’t like about their graduates.  There is a tremendous amount that all faculties can learn about the effectiveness of their programs by talking more to employers – and I mean actually talking, not just having them answer the occasional survey.  So again, why doesn’t this happen?

I think there are two factors behind these phenomena.  The first is that institutions have fundamentally come to conceptualize client feedback in terms of accountability relationships rather than in terms of program improvement.  And the second is that institutions – not unnaturally, I suppose – tend to prefer cheaper data to the more expensive kind.  Students are easy to survey because they’re all around us: employers and alumni give us much more interesting data, but they are harder to reach and the kinds of information that is most interesting require more involved methods of data collection than an email survey.

But it’s not just that better institutions require better data: they need better feedback loops as well.  Precisely because the sector tends to use data for accountability rather than improvement, our systems for improving programs have become impoverished and too often, program-level reviews are box-ticking exercises designed to allow departments to keep doing what they are doing with as little change as little as possible.

That’s not good enough, though.  If we want to drive real change in institutions, we have to develop stronger cultures of continuous improvement.  That’s a process issue as much as a data issue.  And to the extent that current accountability arrangements are getting in the way of us having better data and better processes, institutions really need to think seriously about those accountability arrangements and how they can be changed.