Garbage In, Garbage Out: Nova Scotia Edition (Part 1)

You may have heard of the program costing exercise (as part of an Academic Program Review) that the government of Nova Scotia has foisted on the institutions in that province. Today, I am going to go through how the exercise is being conducted as well as a few ways in which I find it lacking.

Before I start, two nota benes (notas bene?). First, no one has paid HESA to do this analysis. This is a labour of…well, not love, exactly…more like deep irritation. Because second, the contract to run this program costing review—which was administered by the Maritimes Provinces Higher Education Council on behalf of the Nova Scotia government (for reasons that are deeply unclear)—is one that we at HESA Towers bid on and lost to a competitor from the United States (Huron Consulting) who bid about four times higher than we did (yes, really) on grounds that they understand Canadian higher education and costing exercises better than we do.

Bearing that in mind, do feel free to take what follows with whatever grain of salt you want.

Let’s start with the methodology: the basics of Nova Scotia’s approach to this exercise can be seen in this document right here, which I assume is mostly Huron’s Intellectual property with the Nova Scotia government’s logo tacked on. As you can see on the slide I post here, it’s a 7-step process (which you can also see on slide 8 of the linked document if the figure below is too small).

The first thing you should note here is that nowhere in this document is there a definition of the term “program”, which is odd in a program costing exercise. It’s not in section Step One, which is meant to lay out the “academic hierarchy”, not in step 6 where revenue per program is meant to be defined, and not in step 7, where “program allocation” is supposed to explained. It’s just…missing.

This wouldn’t be a problem if programs and departments were 1:1. And of course, in a few cases (notably in smaller institutions, of which there are abundant in Nova Scotia) this is true: the small classics department offers only a minor in classics, the physics department gives out major in physics, etc. Where this gets complicated is when:

• A department offers graduate degrees as well as undergraduate ones or multiple variations of undergraduate programs (major, honours, minors, etc.).
• A department offers more than one degree at the undergraduate level (e.g. a biology department offering degrees in both biology and biochemistry).
• A department offers courses which are part of an interdisciplinary degree.

Now there are couple of ways you can deal with these challenges. One way is just to say “to hell with a program cost review, let’s just look at costs per student at the department level and leave it at that,” on the (probably correct grounds that if the department as a whole is relatively costly, the programs within it are likely costly, too. It also saves a lot of time and effort in data collection and processing, because unlike programs, departments are actually proper accounting units.

The other way to deal with it is the way Nova Scotia did – which is to i) pre-define programs for institutions, ii) tell them to collect data on costs at a course or section level, and iii) tell institutions how to aggregate from the course to the program level (it gets tricky when it comes to courses that can count towards more than one major). It’s a defensible choice, but it comes with some hidden challenges that I outline below.

Steps 2 through 5 of the approach represent an attempt to figure out costs. The idea here is basically to break everything down into “credit hours”.  This means identifying every course and section and the number of credits attached to it, the number of students in each one (credits times students equals “credit hours”) and the “primary” instructor attached to each one (so that appropriate salary costs can be attached to the credit hours produced (this seems like a bit of a nightmare to work out for team-taught courses, but whatever). Then, the direct and indirect course costs for each course are worked out, where direct cost equals faculty pay and benefits and “indirect costs” includes things like materials costs for labs, etc., as well as charges for central institutional support and libraries. 

Now, it is important to understand a couple of things at this point. First, this methodology excludes a significant amount of institutional spending. When it comes down to indirect costs, like physical plant, ICT, external relations and student services are all excluded, for example.

These exclusions aren’t necessarily a bad thing. As long as the same things are being excluded for everyone in an institution, you’re mostly still making apples-to-apples comparisons about which courses cost the most relative to one another even if you’re not capturing every single dollar of overhead costs.  Where one does need to be careful here is that the exclusions probably significantly undercount costs in sciences and engineering (which have higher IT and infrastructure costs that won’t be counted). There are also potentially some issues in making costing comparisons across institutions, because institutions don’t implement the CAUBO definitions that Huron is relying on in a 100% consistent manner. Depending on how IT is managed, for instance, it might show up either as excluded overhead or included in faculty budgets. Probably doesn’t make a huge difference, but worth noting for the sake of completeness.

Then you get to step 6, which is an attempt to allocate resources on a per-program basis. This is definitely more problematic. The formula gives institutions four rules to work with for inclusion:

1. Tuition and program/course fees, aggregated by program, based on the student’s program of enrolment.
2. Funded scholarships, bursaries, and financial aid to be included and only if it is recognized as revenue in financial statements and can be attributed to the specific student or specific program.
3. Provincial grant amounts, but only if they are clearly targeted to a specific program  (!!!!!!!).
4. For students enrolled in multiple programs, the primary program of record will be allocated 100% of both revenue and costs. Institutions have discretion to decide what program they record as “primary.”

Well now. 

This definition excludes significant income. Obviously, it excludes things like research funding (which is ok, because professorial research time is excluded on the other side of the ledger), but more importantly, it excludes the basic government grant and revenue from any students who have not declared a major (which is first-year students in several but not all programs, and in some cases, second-year ones as well). If the effects of these exclusions were equal across the board, that would be one thing. But they are not. 

It is certainly possible to produce program costing numbers this way. It’s just that after a certain point, it’s not clear why you’d want to do it. There are indeed difficulties in trying to allocate every dollar in and out of a university, but if the data is more reflective of the choice of methodology than of the actual underlying reality, why would anyone want to use that data to determine policy? 

(To be fair, even a crap methodology will still find the programs that are making or losing money at an egregious rate. If that’s all you want to achieve, you could do that just by looking at faculty: student ratios. To take a completely random example: if your music department has more faculty than students then it is losing a lot of money, full stop. There’s no need to go through the extensive, painful data collection to learn that.)

I’m not close to done with this analysis – trust me, there is one humdinger of a methodological atrocity still to come – but I’m already sitting at over 1350 words and you need a break. Come back tomorrow for the exciting conclusion.