Older readers may remember a character created by the late Peter Cook called Sir Arthur Streeb-Greebling. Sir Arthur had spent many years attempting to teach ravens to fly underwater, so far without any success. Interviewed on television, he remained fully confident of future success: ¡°I¡¯ve learned from my mistakes, and I¡¯m sure I can repeat them.¡±

In 2015, Times ÖйúAƬ published an article of mine describing the many problems with the teaching quality assessment exercise that ran in England between 1993 and 2004. Like the teaching excellence framework, the aim of the TQA was to provide information about the quality of teaching across the sector, subject by subject and institution by institution. This would guide choices by students and funders (through the block grant: this was before significant tuition fees were introduced) as well as incentivising institutions to raise their teaching game and thus lift standards of teaching and education across the system.

However, there was little evidence of any of these objectives being achieved and the process died quietly during one of the government¡¯s deregulatory phases. In the article, I wondered if the TEF would suffer the same fate, although I was sure that the department would have ensured that the relevant minister was well apprised of what had happened before and of what was needed to make the new process more robust.

Not the least of the difficulties with the TQA was the lack of consistency of outcomes, both between subjects and, more damagingly, between institutions. This was partly about the complexity of the judgements the assessors were having to make, and partly about the virtual absence of any effective moderating mechanism (indeed, a senior manager of the process once told me that they did not even have time to read the reports properly before publication). Unfortunately, it is hardly a surprise to learn, from a ¡°quick and dirty¡± sampling of institutions¡¯ experiences (and from Simon Baker¡¯s detailed analysis), that precisely the same questions are arising with the TEF.

ÖйúAƬ

There seem to be four sets of issues: the statements of findings evince a wide range of views of what is meant by excellence; there is little consistency in how the statements reference the extent to which provider submissions mitigated metrics; there are some cases in which the assessors¡¯ understanding of metrics was clearly insufficient for them to be able to make a proper judgement; and very similar evidence has resulted in different awards.

Of course, the TEF is only in its early stages. But with full-cost fees and student debt, the stakes are far higher than they were with the TQA. Student recruitment is becoming increasingly challenging, and use will be made of TEF awards by Ucas and by institutions that have done well out of the process, irrespective of the misgivings of the sector.

ÖйúAƬ

It is imperative that these issues are tackled. Until they are, the TEF cannot be regarded as a safe guide even to the very limited range of information about student education that it covers, let alone to the quality of teaching across the sector.

Roger Brown is the former chief executive of the ÖйúAƬ Quality Council.

Read next: more TEF news and analysis