HESA

Higher Education Strategy Associates

Tag Archives: Testing

January 11

Admissions policies: Marks-Based, Broad-Based, or Random?

Though here in egalitarian Canada we don’t like to talk about it much, the fact of the matter is that universities are selective.  More people want to enter them than there are places available.  The more prestigious the institution, the greater the imbalance between demand and supply of places, thus requiring more challenging and discerning barriers to entry (though self-selection reduces actual application numbers somewhat).  The question is: on what basis should we select students?

(OK, some of you are now saying “not so fast! not all universities are selective!  What about countries like France or Germany which give automatic access to everyone who gets a Baccalaureat/Abitur? which have “open access”?  Or what about Quebec?”  Well, in fact “open access” countries are nothing of the sort – they just put the selection filter further back in the educational chain when they stream kids at age 12 or so.  Quebec is a different case: the UQs will accept anyone in possession of a CEGEP diplome d’etudes collegiales (DEC) which in global terms is pretty radical.  But selection still exists at the rest of the province’s universities).

Now, of course, in selecting students, everyone thinks we should consider “merit”.  But in most of the world, merit simply means “taking exams well”.  It means passing a set of secondary matriculation exams (e.g. in France and Germany), or a set of national university entrance exam (China’s gaokao, College Scholastic Ability Tests in Korea, etc) or even in some cases specific university entrance exams (for instance, the University of Tokyo – an exam sufficiently difficult that specially-programmed AI robots cannot yet pass.  Occasionally, as in the US or Sweden, psychometric exams like the SAT are used as well.  Are these methods fair?  Depends on your criteria.  If you think test and exam-testing are the be-all and end-all of merit, then yeah.  If not, no.  But finding alternatives is tricky.

Famously, the elite US universities went for a broader definition of merit in the 1920s, one which emphasized character and sporting ability.  Of course, the reason they did this was because their WASP donor base was getting pretty freaked out about the number of Jews getting in under the old scholastic-ability-only rules (see Jerome Karabel’s The Chosen for more on this).  That worked until the late 1960s/early1970s, when growing concern about racial inequality led some to start musing about whether elite private institutions shouldn’t be forced to accept more minority students.  Lo and behold, the definition of merit was changed to avoid clubby, exclusionary things like “character” (at least in the clubbable sense of that word) and include nice things like “diversity”.  Of course what it didn’t do was restore points for simple “academic merit” alone.  Nowadays, some see this as discriminating quite significantly against one ethnic group in particular: Asian-Americans, who by some reckonings are estimated to have a 67% lower chance of admission that a white student with similar GPAs/SATs.

In Canada, we’ve mostly relied on a portfolio of marks over high schools rather than a set of exam results, but the result is pretty similar: the academically inclined (a status yuuuugely-correlated with parental education levels) win out just the same.  Because none of our institutions is that selective, we’ve never seen the kind of crazy admissions scenes the US has, but a few selected hard-to-enter faculties have, most notably the Sauder School at UBC.  Back in the early 2000s, it took averages in the mid-90s or higher to make it in.  But the business community who hired Sauder graduates wasn’t enthused about the quality of the output: too many kids who knew math, not enough who understood leadership.  So Sauder moved to something called broad-based admissions, which basically meant a more intensive evaluation of students in order to create an entering class which was less academically focussed and more “well-rounded”.  Not surprisingly, some think this gives an edge to the white upper middle class and served mostly to reduce the number of Chinese students at Sauder (which, say it softly, may have been what the Vancouver business community meant when it said it wanted fewer kids who were “good at math”).  Yet broad-based admissions were such a success that they were introduced across the university just a few years later.

Now there are ways to run broad-based systems which don’t simply reinforce cultural capital:  the Loran Scholarships have a long track-record in doing precisely that, mainly by evaluating achievement in the context of parental background.  But most systems don’t do that, and as the University of Manchester’s  Steven Jones’ has pointed out in a couple of excellent recent articles, most attempts to broaden the base of assessment end up reinforcing privilege.  Which leaves you with a conundrum.    If you set a firm marks-based standard, you’re probably giving a huge advantage to those with better-educated parents; in a broad-based system, you’re probably giving an advantage to those with a lot of cultural capital.

Is there another way to do it?  Well, yes.  Two, as a matter of fact.  The first is to try to select on something other than academics or character.  Robert J. Sternberg, an American academic, has written an engaging book entitled College Admissions for the 21st Century which recounts his own efforts to create tests to complement the SAT/ACT by measuring things like tacit knowledge, wisdom, and creativity.  Some skepticism is warranted – Sternberg is talking his own book, after all – but it’s an intriguing effort that more should emulate.

The second way is lotteries, which have been used extensively for medical school admissions in both the UK and the Netherlands (though it is being phased out in the latter).  Usually what happens in admissions lotteries is that the bar for admission to the lotteries is set substantially below what it would be if pure competition were allowed to reign.  So, if we take the case of somewhere like the Sauder School, instead of setting the bar at a 95 or 96% average, you set the bar at say 80%: not so low as to let in just anyone, not so high as to exclude candidates who might really benefit from a Sauder education.  Maybe that gives you five times as many students as you can handle: fine, just pick one out of five of these students randomly.  In the Dutch variant, you might give a bit of an edge to higher-scoring students by giving them multiple entries into the lottery, but that’s optional.

Clearly, this doesn’t give you “the best” students, if you define “the best” as doing well on exams or being elected student council President.  But that’s the point.  It gives you a good class of students without creating educational arms races which produce either the gruesome test-taking cultures of East Asia or the nauseating college admissions industry of the US.  As such, it deserves to be in wider use.

October 05

A Brief History of Exams

Written exams are such a major part of our schools and universities that we forget sometimes that they are not actually native to the western system of education.  How did they become so ubiquitous?  Well here’s the story:

Originally, the Western tradition eschewed exams.  Universities offered places based on recommendations.  If one could impress one’s teachers for a few years, one might be invited to audition for right to be granted a degree. In medieval universities, for instance, one obtained a degree once one was capable of giving lectures or credibly argue a particular position in a debate format (the disputatio).  This was more or less the case right through until the 17th century.  This was completely different from how it was done in China.  There, ferociously difficult examinations for entry into the Imperial Civil Service had been the norm since the first century AD (give or take a couple of centuries of inter-dynastic interregnums due to societal collapse, civil wars, etc).  To help students through these exams, “academies” were created, which, with a bit of squinting, can be seen as forerunners of today’s universities (for more on early Chinese higher education see here).

In the late 16th century, a Jesuit priest named Matteo Ricci was sent to China and eventually rose to a very senior position within the order.  He was very impressed by the competitive and meritocratic nature of the Chinese examination system, and described it in glowing terms to his superiors in Rome.  Being a pedagogically-minded order, the Jesuits themselves adopted written examinations in order to make their own system tougher and more competitive.  In the 18th century, absolutist reformers trying to create meritocratic civil services (as opposed to ones run by aristocratic place-holders) decided to put the Jesuits’ “Chinese” system to work.  Starting in Prussia, then spreading around Europe over the following century, bureaucrats now had to pass examinations.  As more and more people tried to apply to the civil service, the universities – which were mainly prep schools for the civil service – became more crowded and gradually introduced their own entrance examinations as well.  The first of these was the German Abitur, which is still the qualification required to enter university.

The question of who set these exams – the education ministries in charge of secondary education?  the universities themselves? – was answered different ways in different countries.  In the United States, the Ivies maintained their own exams well into the twentieth century.  To keep out the riff-raff they would do things like test for ability in Greek – a subject not taught at public schools.  As universities began to expand the range of their intakes, they started to see problems with exams based on curricula and started looking for something that would measure potential regardless of which state or school they came from.  This led them to consider psychometric examinations instead, and hence the SAT was born.

Psychometric testing never really caught on outside the US (thought Sweden uses a variant of it).  Generally speaking, the dominant form of testing around the world remains a high-stakes test at the end of secondary school: the gaokao in China, the Korean suneung and the Japanese center are the most famous of these, but most of Europe and Africa operate on some variant of this (albeit without causing the same level of commotion and stress because European university systems are less hierarchically stratified than East Asian ones).  In many of the post-Soviet countries, university entrance exams were a source of lucre.  A prestige institution could set its own exam, and rake off money from students either through preparatory classes or by requesting bribes to pass.  The establishment of national university entrance exams in these countries were thus as much as an anti-graft measure as a pro-merit measure.

Many parts of the world – but particularly Asian countries – are seeing the downsides of basing so much on a single set of exams, and are trying in various ways to try to de-emphasize testing as a means of distinguishing between students, both because they are seen as overly stressful to youth and because the results have been time and again to reinforce class privilege.  The problem with the latter is that no one has yet come up with alternative measures of academic prowess or potential which are significantly less correlated with privilege; and exam results, whatever their faults, do provide transparency in results, and hence a greater appearance of fairness.

In short: there’s lots wrong with high-stakes exams, but they aren’t going anywhere soon.

March 14

Canadian Higher Ed Exceptionalism, Part 1 (An Occasional Series)

For awhile now, I’ve been writing about other national systems of higher education in our, “Better Know a Higher Ed System” series, in part to throw Canada’s own policy system into sharp relief. But sometimes it’s better to look at some things a bit more directly, so today I want to start exploring some areas where Canada really is an exception, globally.  And there’s nowhere we stick out more than in the way we admit students to university.

There are a limited number of ways to admit people to universities.  One of the most common is simply to use the scores from a common secondary matriculation exam as the basis for admission decisions.  Most of la Francophonie works this way, since they’ve all modelled themselves on France’s Baccalaureate.

Another option is to have a national university entrance exam, separate from matriculation.  The most famous of these is China’s gaokao, which draws on a millennia-long Chinese tradition, but which is in fact only 35 years old, and a product of Deng Xiaoping’s post-Mao reforms.  National exams are often a response to widespread cheating and corruption in schools.  In 2004, Russia introduced a new national exam with heavy security measures specifically to try to weed out academic corruption (it was only partially successful – since getting into university is a way for Russian males to avoid the draft, the impetus for academic corruption is pretty powerful).

In some places, individual universities have their own entrance exam, though these tend to exist only where a national exam is already in place.  Japan and Romania are two examples of this: in both countries, the more “elite” universities (e.g. Tokyo, Kyoto, Politehnica, Bucharest) have chosen to ditch the national exams and establish their own, for reasons of prestige, if nothing else.  And then, finally, you have the American options – not a university entrance exam but a national aptitude test, such as the SAT or the ACT.

So, now imagine trying to explain to foreigners how students get accepted to university in Canada.  Only Alberta, with its matriculation exams, has anything like the kind of standardized testing seen almost everywhere else in the world. In the rest of the country, you’re admitted entirely on the basis of grades based on high school marks predicated, to a considerable extent, on work portfolios rather than exams, and where grading standards between schools are only loosely consistent.  To the extent that there is fairness at all, it comes through the informal judgement of hundreds of admissions officers who, through simple experience, “know” which schools are easy graders, and take this into consideration when awarding places.

From the perspective of most other countries, the Canadian approach looks like sheer lunacy.  The scope for corruption in our system is enormous, but it’s simply not an issue here.  Everyone accepts the professional judgement of admissions officers and there are few complaints.  Such deep trust in the system is what has spared this country (outside Alberta, anyway) the kind of high-stakes exam nightmare that Americans endure.

In short, one thing that makes Canadian higher education exceptional is trust. That’s great, but trust is fragile.  It’s not something we should take for granted.

October 14

Teaching, Testing, Grading

In the last couple of months, some very interesting memes have started to take shape around the role of the professoriate.

Grade inflation – or grade compression as some would have it – is of course quite real. Theories for it vary; there’s the “leave-us-alone-so-we-can-do-research theory,” and also the “professors-are-spineless-in-the-face-of-demanding-students theory.” Regardless of the cause, the agent is clear: professors. They simply haven’t held a consistent standard over time, and that’s a problem.

About two months ago, the Chronicle put together a very interesting article on Western Governors University and how they’ve managed to avoid grade inflation. Simply put: they don’t let teachers grade. Rather, they leave that job to cohorts of assessors, all of whom possess at least a Master’s degree in the subject they are grading, and who are quite separate from the instructors.

This kind of makes sense: teachers are subject matter experts, but they aren’t expert assessors, so why not bring in people who are? Unlike professors, who have to put up with course evaluations, independent assessors have no incentive to skew grades.

One could take this further. Not only are professors not experts at grading, but they aren’t necessarily experts at devising tests, either. Solution? Step forward Arnold Kling of George Mason University who recommends improving testing by having outside professionals taking a professor’s lecture notes and course readings and fashioning a test on the basis of them.

Are there good reasons to try these ideas? On grading, the gains might be on quality rather than cost. Informally, TAs do a lot of the grading in large classrooms anyways so it’s not as if we aren’t already quasi-outsourcing this stuff. But the TAs have no more expertise than professors in terms of assessment, so professionalizing the whole thing might be beneficial. On testing, you might not get cost advantage unless you had some economies of scale (i.e., you’d need multiple participating institutions to make it worthwhile), though again there may be quality advantages.

Of course, to get any cost savings at all on either of these, you’d need to get professors to explicitly trade their testing and marking responsibilities in these areas for greater class loads. Have them teach three courses a term instead of two, but do less in each of them. It’s hard to say if anyone would bite on that one; but given coming funding crunches, it might be worth somebody at least trial-ballooning these ideas during their next collective bargaining round.