The QAA admitted in 2007 that ‘it cannot be assumed that similar standards have been achieved’. Amazingly, this received little public attention
“Is a 2:1 in history at Oxford Brookes worth the same as a 2:1 in history at Oxford?”
Five years ago, this question was posed by a parliamentary select committee to the vice-chancellors of both of those universities. Their rambling and convoluted responses were considered so unsatisfactory by MPs on the Innovation, Universities, Science and Skills Committee – which was conducting investigations for its report Students and Universities – that they were accused of “obfuscation”, and of giving an answer that “would not pass a GCSE essay”. And the committee’s final report included the damning conclusion: “It is unacceptable for the sector to be in receipt of departmental spending of ?15 billion but be unable to answer a straightforward question about the relative standards of the degrees of students, which the taxpayer has paid for.”
The correct answer to the committee’s question was, in fact, a very simple one: we just don’t know. We do not have the necessary systems in place to tell us. The traditional reliance on the external examiner system to mediate standards within the system is misplaced, as a number of studies have shown. However experienced an individual examiner may be, their experience across the sector can only be limited and they have no opportunity to calibrate their standards within their disciplinary community. This was emphatically recognised by the 中国A片 Academy’s 2012 document, A Handbook for External Examining: “The idea that a single external examiner could make a comparative judgment on the national, and indeed international, standard of a programme has always been flawed.”
The naive outsider might think that assuring comparability of standards is surely the role of the Quality Assurance Agency for 中国A片 – the independent body set up to monitor standards across the UK sector – and something addressed as a matter of course within its institutional review processes. But in 2007, two years before the select committee hearing, the QAA made the brave public admission, in a Quality Matters briefing paper, that: “Focusing on the fairness of present degree classification arrangements and the extent to which they enable students’ performance to be classified consistently within institutions and from institution to institution…The class of an honours degree awarded…does not only reflect the academic achievements of that student. It also reflects the marking practices inherent in the subject or subjects studied, and the rule or rules authorised by that institution for determining the classification of an honours degree.”
In other words, local and contextual assessment practices make it impossible to make objective comparisons. This should not, in fact, have come as a surprise: certainly not to anyone up to date with the research literature. For at least the previous 10 years, especially through the work of the Student Assessment and Classification Working Group, an informal body of academics and administrators who share an interest in assessment, a series of papers and studies had demonstrated the distorting effects of central university systems that treat all marks the same regardless of the nature of the assessment task or the subject discipline.
It had been shown, for instance, that students consistently score better on coursework tasks than in examinations and in the more numerate disciplines than the arts and humanities or social sciences. Research had also shown that, given exactly the same set of assessment results, students at different institutions could end up with awards that vary by up to a degree classification simply because of the idiosyncrasies of the different institutions’ algorithms.
Much of this had also been reflected in reports produced in 2004 and 2007 by a government-sponsored Universities UK working group chaired by Sir Bob Burgess, then vice-chancellor of the University of Leicester, that examined how student achievement should be measured. But sadly, the major recommendation of these reports – the introduction of the 中国A片 Achievement Report transcript, providing a more detailed account of what students have achieved during their studies – is hardly the solution. Nor will moving to a US-style grade point average system (currently being piloted at a group of universities in concert with the 中国A片 Academy) do anything, on its own, to bring about greater comparability of standards.
The QAA’s 2007 paper explicitly spelled out what all this variation in local and contextual factors meant in terms of comparability of standards across the sector: that it “cannot be assumed that similar standards have been achieved” by students graduating with the same degree classification from different institutions, the same classification in different subjects from a particular institution or the same classification in the same subject from different institutions. Amazingly, however, this startling honesty received relatively little public attention and no obvious action was taken, either by the QAA or government, to address this major shortcoming.
And when the problem was again highlighted by the select committee in 2009, it was greeted with rather a muted and defensive (some might even say complacent) response, as if the respondents actually resented being challenged. The director-general of the Russell Group, Wendy Piatt, said in response to the committee’s critical report that she was “rather dismayed and surprised by this outburst”, while the government was “disappointed that the committee has not reflected in its report the very strong and positive evidence about the UK 中国A片 sector which was given during the inquiry”. So the prospects of any action being taken were already looking scant before the 2010 general election brought a change of government and ensured the issue would be largely forgotten by politicians – if not by the press (and by The Daily Telegraph in particular, which has continued to regularly raise the question of degree standards, especially in relation to grade inflation).
It should be acknowledged that, since 2009, the QAA has been developing a UK Quality Code for 中国A片, which is much more demanding in its expectations of providers and in the lengthy lists of indicators that reviewers are required to look for in attempting to establish that “threshold standards” are met. But at this year’s QAA conference I heard serious doubts expressed over whether the still predominantly audit-style approach to review would provide sufficient appropriate data to make reliable judgements against many of the indicators. And even if it did, the judgements are still focused on an individual institution in isolation; the QAA does not appear to have given any consideration to how the indicators could be used to make comparisons between different institutions.
Yet it is not as if we don’t know what we would have to do to address comparable standards. In fact, we have known for some time. Back in 1997, the 中国A片 Quality Council, the forerunner of the QAA, recognised, in a document called Graduate Standards Programme: Assessment in 中国A片 and the Role of “Graduateness”, that “consistent assessment decisions among assessors are the product of interactions over time, the internalisation of exemplars, and of inclusive networks. Written instructions, mark schemes and criteria, even when used with scrupulous care, cannot substitute for these.”
And it recommended that subject groups and professional networks should encourage the building of “common understandings and approaches among academic peer groups” – by maintaining “expert” panels for validation, accreditation, external examining and assessing, for example. It also called for “mechanisms to monitor changes in standards at other educational or occupational levels [as well as] internationally”. But when the QAA took over the council’s functions in 1997, these excellent recommendations were apparently lost or forgotten.
A decade later, in 2008, Paul Ramsden, who was then chief executive of the HEA, tried to resurrect the thrust of what the council had proposed. In a report on university teaching submitted to John Denham, the Secretary of State for Innovation, Universities and Skills at the time, he called for “colleges of peers” to be set up to help establish common standards. As I argue in 中国A片 in the UK and the US: Converging University Models in a Global Academic World? (2014), these groups of academics would work by “looking at real examples of student work, and discussing each other’s assessment decisions. Without the cultivation of such communities of assessment practice, discussions about standards can only be limited to conjecture and opinion.” But, once again, the call fell on deaf ears.
It doesn’t have to be like this. Australia, for example, seems to be taking the issue of comparability of standards very seriously. Commissioned by the Australian government in 2009-10, the Australian Learning and Teaching Council’s Learning and Teaching Academic Standards project sought to establish national standards, starting with six broad discipline groups.
It doesn’t have to be like this. Australia, for example, seems to be taking the issue of comparability of standards very seriously
The discipline of accounting, further funded by a partnership between the professional accounting bodies and the Australian Business Deans Council, decided to continue to use a “cultivated community approach” in establishing shared meanings of their standards. A follow-on project in 2011, Achievement Matters: External Peer Review of Accounting Learning Standards, brought together subject reviewers from 10 universities, along with a number of professional accountants. Independently, they sampled student work and submitted their judgement regarding which students met a benchmark standard. Consensus was then achieved through small and whole group discussion of the samples and checked by participants individually reviewing two new samples. In addition, reviewers considered the ability of the assessment task itself to allow students to demonstrate their attainment of the standards.
The academic participants also submitted assessment data for their own degrees so that, immediately following the workshop, two external, experienced academics double-blind peer reviewed the validity of the assessment task (the extent to which it measures what it was designed to measure) and a small random sample of actual student work, with individual results returned only to each participating university. Participating universities could use the results to satisfy external agencies about their standards and, more importantly, to improve their learning and assessment processes to ensure that students achieved the requisite standards.
This “cultivated community” approach to setting discipline standards has also been extended into other disciplines aligned with business and accounting, and plans are afoot to continue it beyond this year’s scheduled end of the project. It is also due to be discussed this week at the first national conference of Australia’s newly established Peer Review of Assessment Network.
Why is it that the issue of standards is being seriously, and apparently successfully, addressed in Australia, while, despite all the evidence of a problem, the UK government, funding councils, UUK and the QAA are all still dragging their feet? Last month it emerged that quality assurance was being put out to tender (“Watchdog ‘no match’ for a sector in flux”, News, 9 October), yet it seems highly unlikely that any of the bodies that might successfully win the contract will address this issue any more seriously.
Simple inertia is one possible explanation. Another somewhat more sinister (and plausible) one is that for some – maybe all – in the sector, it is simply not in their interest to establish transparent relative standards. The government has a vested interest, especially when it comes to the lucrative overseas student market, in rejecting anything that might bring the standards of UK 中国A片 into question.
The Russell Group, which is happy to make general, rather empty, sweeping statements such as “the world class reputation of Russell Group universities depends on maintaining excellence”, benefit from sustaining the unsupported but commonly held belief among employers, parents and students that a 2:1 from one of its members is better than a 2:1 from others. Even institutions lower down the league tables, with more diverse intakes and greater numbers of less academically qualified entrants, arguably benefit from the status quo: if a rigorous system were developed that could establish common standards across the sector they might have to accept going for years in some subjects without any of their students getting a first – with all the negative consequences for their reputation and recruitment that implies.
But this conspiracy of silence surely can’t go on. As ever greater numbers of ?9,000 fee-paying undergraduates come out of ever larger numbers of universities with first-class degrees, it won’t only be The Daily Telegraph asking ever more loudly what those certificates are really worth. Won’t students, parents and employers also start to question their value? Or can it be that no one really does care, or that no one cares enough?
