For a sustained period in the 1960s, Harvard University had to cope with what today would be called a problem of human resources.

A member of its philosophy department was noticeably unproductive. Years came. Years went. The arts faculty began to grumble. Their philosophy colleague¡¯s lack of scholarly activity was, they felt, unacceptable in a modern university. The corridor talk grew; early whispers transmuted into a desire for the university to face the fact that this man was now, pretty plainly, a dud.

The university considered the matter, weighed people¡¯s concerns and resolved to demur. It was not right, the institution decided, to bend to gossip or attempt to sanction this apparently spent or lazy philosopher. The dean persisted in paying him the usual wage increases. The choice was made to get on with the everyday business of allowing academic freedom. People thought that was the end of it; they were stuck with him.

One day, in the cold winter of 1970, something unusual happened. To the faculty¡¯s surprise, the old dud turned in a piece of work. If nothing else, it was agreed at the time, his manuscript¡¯s title had the merit of pithy brevity. Then everyone went back to their corridors and their lives.

ÖйúAƬ

That manuscript¡¯s title was just four words long: A Theory of Justice. The scholar¡¯s name was John Rawls. Today, according to Google Scholar, A Theory of Justice has been cited 44,000 times.

My friend Henry Rosovsky told me and my husband, Andrew Oswald, about this event, which took place a few years before Rosovsky became dean of the Faculty of Arts and Sciences at Harvard.

ÖйúAƬ

Rosovsky is a consummate statesman; he has absorbed much in his rich life and has put that wisdom to use in ways at which we can barely guess. (I highly recommend his book, The University: An Owner¡¯s Manual, to all would-be university leaders.)

His memorable story about Rawls led me to have a heated midnight discussion with my husband about the research excellence framework and whether it is making ÖйúAƬ in Britain better or worse.

The story led us to ask: could Rawls¡¯ A Theory of Justice have been published under the restrictions of our country¡¯s REF?

Amanda Goodall

Governments do not understand, and cannot motivate, the iconoclastic researchers of the world¡­The great scholars do their work because it is burning inside them

Andrew

I do not believe Rawls¡¯ book would have survived the world of the REF. The REF is inimical to originality. It fosters quantity. Especially with regard to the future, I am doubtful of the claim that our country benefits from it. The whole issue reminds me of an occasion when I and a few other UK professors were summoned to the room of a famous politician. ¡°How should the ÖйúAƬ sector be run?¡± he asked us. I suggested that we might want to leave it to scholars to decide. He replied, ¡°But we spend all this public money. We can¡¯t possibly do that.¡±

Let me mention a natural, although perhaps unconventional, reason for my scepticism about research assessment exercises.

In 1979, in anguished nervousness, wearing a faded medium-length black gown and a mildly wilting white bow tie, I entered a half-lit room in the centre of Oxford to go through my doctoral viva in the subject of economic science. There were only two people in there. They sat at the far end of the room, staring at me from the other side of an old brown desk. Both, in the sheen of their black and royal blue robes, had a slightly damp air, presumably from the rain of that very English afternoon.

One was a man who appeared to me to be close to 100 years old (he was actually only close to 80). Over half-moon glasses, he looked directly into my eyes. I had never met this man, nor even seen him, before. His name was John Hicks. Slowly, almost painfully, he leaned forward. He peered intently at me across the vast desk. It was impossible to tell what he was thinking. Then, very quietly, he asked an incredibly difficult opening question (¡°What about the traverse, young man?¡±). One reason I was almost paralysed was that this was the first time I had been in the presence of someone as famous as he was (he turned out to be a decent person, with a slight stammer and a gentle disposition - and, even better, I eventually realised that I was probably going to pass this damned examination).

ÖйúAƬ

The reason he was known to me was that, in 1972, when I was barely out of high school, Hicks had won what is called, slightly inaccurately, the Nobel Prize for Economics, for inventing impossibly huge chunks of the discipline in which I was striving to become a doctor. He had come up with a number of the equations used in my thesis.

Yet how many research assessment exercises had there been in this man¡¯s lifetime? None.

How many government-mandated assessors had examined the work of Hicks? None.

How many of the hundreds of thousands of Swedish kronor for the Nobel prize did he keep? None (he gave them all to charity).

If, at the end of that DPhil viva, I had had the temerity to ask him for reasons why our nation needed a government-run research assessment exercise, I feel reasonably confident that I could guess his answer: there were none.

The perplexing truth is that governments do not understand, and cannot motivate, the iconoclastic researchers of the world. They walk to a different beat. The great scholars do their work because it is burning inside them and because it is simply those individuals¡¯ destiny, beyond even their own power to control. You may as well tell Picasso not to paint or Einstein to stop scribbling and get on with his clerical duties. A REF cannot stop these people. Or start them.

Amanda

I think the REF has done some good things.

At the time of its publication, Rawls¡¯ A Theory of Justice was hailed in The New York Review of Books as ¡°the most substantial and interesting contribution to moral philosophy since the [Second World War]¡±. Across all his work, Rawls has been cited more than 80,000 times. Arguably, his justice work might not have been published under the REF, nor under its predecessor the research assessment exercise, primarily because of the time it took Rawls to construct it. And of course he would have had only one piece of work to submit, instead of the required four.

But let us not forget what Rosovsky said: first, the Rawls example is extremely rare, as with Hicks. And second, Rawls went through a tremendously challenging tenure process. Selection for UK and European universities has never been as tough as going through the tenure process in a US research university - at least it certainly wasn¡¯t in the 1980s, prior to the introduction of the RAE. This and its spin-offs have toughened the hiring process and in some ways forced it to be more open. As a woman, I feel pleased that the criteria for getting a job in academe are more transparent now.

Another important issue related to the Rawls example is leadership. In US universities, especially the private ones, presidents have traditionally had more direct powers to control standards than in UK and European institutions. This has changed dramatically in the UK, and increasingly so on the Continent. The RAE had a lot to do with these changes, because it gave vice-chancellors ammunition from an exogenous framework that had to be adhered to. Therefore, university heads were justified in wanting to have a greater say in who became pro vice-chancellor or head of department and, importantly, which academics got jobs.

Andrew

It is true that I am suggesting something a little counter-intuitive. The idea of having a REF is like the idea of having government inspectors for butcher¡¯s shops and for schools. It just seems obvious sense, to a managerial type, because after all we need to ensure quality and value for money for the public. However, I do not believe this logic. Researchers are best left to themselves. The power of competition will itself lead to world-class quality.

Amanda

Researchers should be ¡°left to themselves¡± only if they can be trusted to do the right thing. I will give you an example taken from Clark Kerr¡¯s autobiography The Gold and the Blue (volume one). Kerr was the first chancellor of the University of California, Berkeley, in 1952, and later president of the University of California, from 1958 to 1967. He was a distinguished economist and many attribute Berkeley¡¯s outstanding success to the standards he put in place when he was chancellor. Kerr was a fervent supporter of selecting notable scholars as department chairs. When he was chancellor of Berkeley, he fought hard to gain the executive power to choose department heads. A quote from this period is poignant: one head told Kerr that ¡°all departments had an unalienable right to be no better than they wanted to be¡±. Kerr responded that he only ¡°accepted that ¡®right¡¯ for departments in the very top rank¡±. And that is the point. If you want to leave researchers to themselves, it should depend on which researchers.

There are many examples of old boys¡¯ clubs with members who select others to join them - generally men who look and sound just like themselves. How long does it take for the selection to lead downwards? After all, why would a member want to invite someone who is more talented than they are to join the club? Her success might reflect badly on his performance. This is why the hiring process in the best universities is monitored very closely - to ensure that departments do not select down to a comfort zone.

Andrew

Our main intellectual competitor in the world is the US. Does that competitor use a REF? Of course it does not. So why should we?

Amanda

First, the RAE and REF are really just glorified rankings. It is fashionable to criticise all rankings in the world but such a position is laughable. I usually ask people who attack university rankings: what do you do when you want to buy a car? Or a fridge? Or when you have to find a school for your children? Or when you are looking for a hotel? You go to Which? or What Car? or some other heuristic measure to help you make the right choice. So why pretend that universities are any different?

Second, the top US universities may like to give the impression that they are above rankings, but if (God forbid) they slipped down the list, we would really see some action. Rankings started in the US in the 1900s. Indeed, when Kerr was chancellor of Berkeley he used the rankings produced by the American Council on Education to motivate staff. Kerr¡¯s yearning was to overtake Harvard, Yale University, Princeton University and other top US institutions - a desire that was eventually met in 1964 when Berkeley was ranked as number one.

Third, you may have an idealised view of American research universities, if Benjamin Ginsberg¡¯s The Fall of the Faculty is correct. He bemoans the rise in the number of administrators and suggests in uncompromising terms that ¡°many of the non-academic administrators are career managers who downplay the importance of teaching and research¡±.

ÖйúAƬ

Ben Martin, professor of science and technology policy studies at the University of Sussex, has done a lot of work on assessment exercises. He believes that in the early rounds of the RAE there were substantial benefits (for example, in terms of more emphasis on research, clearer institutional research strategies and better research performance), while the costs were relatively small. But as the RAE became more important and more sophisticated (not least in response to earlier criticisms by academics), the burdens and costs associated with it rose, while the benefits were inevitably subject to diminishing returns.

Andrew

Martin also raises a number of likely problems with the impact measure - we have not even talked about those yet. He suggests that the attempt to measure impact in the REF is laudable but flawed because there is no recognition that impact is multi-dimensional in nature. He worries that the methodology for assessing impact is likely to be criticised when the REF results are announced.

[Research assessment exercises] have toughened the hiring process, and in some ways have forced it to be more open. As a woman, I feel pleased that the criteria for getting a job in academe are more transparent now

I accept that having a REF will lead to greater total research output in a country. But is that really the appropriate criterion?

As the Soviet Union discovered, if you set a planning target of 10,000 tractors then you will get that many tractors. The problem, unfortunately, is that common sense and quality get thrown away in pursuit of the numbers; many important things are neglected; the standard is adjusted down to meet the target; and the consequences of the neglect of broader issues are then deliberately hidden from observers¡¯ view. In the case of the REF, everyone in a British university feels under pressure to produce four refereed journal articles in six years but nobody feels pressurised to be a breathtaking first-year lecturer, or an entertaining speaker at alumni events, or a diligent essay-improver of final-year undergraduates, or a wise-headed sage on university committees that allocate millions of pounds. A REF makes it necessary for all scholars¡¯ efforts to be pushed through the eye of a needle.

Amanda

I do think the RAE was very useful to the UK system, for the reasons I have outlined, but I agree that there are problems with it. It is a managerial tool that has become far too bureaucratic.

So, we agree that we want governments to fund high-quality research. How might we redesign the system?

Andrew

A reasonable question to ask is: if the government is intent on having a REF, what kind of design should be followed?

My colleague Daniel Sgroi and I have tried to tackle this issue in an article ¡°How should peer-review panels behave?¡±, which will be published in a forthcoming edition of The Economic Journal. Our argument is that peer-review panels should use a mixture of citations and journal rankings (with a dash of experienced subjective judgement as well). The weights on these two, we argue, should vary in what could be called a Bayesian way.

In plainer English, the idea is this: when a published article has been out only for a year or two, a review panel should start by assuming that it is as good as the average quality of articles published in that particular journal. Over the years, it becomes possible to measure the number of citations that are accruing to the article. As this extra information becomes available to a reviewing committee, they should gradually put more weight on the number of citations and place less importance on the rank of the journal.

Yet, perplexingly, the current REF rules specify that no committee is allowed to use journal rankings. Many subpanels have also deliberately turned against all use of citations data. Neither of these things makes sense (at least to me).

Amanda

I agree that that seems madness. A government report recently presented evidence showing that there is little difference in outcome when the measure for allocating research funding is based on research council income or the RAE (see ¡°Alternative to REF delivers similar result on QR funding¡±, Times ÖйúAƬ, 11 April).

Scholars such as Charles Oppenheim and the University of Lancaster¡¯s Jim Taylor have produced convincing evidence of the value of citations data. Oppenheim questions why universities and other bodies should go through the anguish and expense of research assessment exercises when readily available statistics give virtually the same results. He says that even for the social sciences and humanities, the primary predictor should be citations and/or research council funding, moderated by human intervention. For the sciences, Oppenheim believes that citations and research council funding should be the only factors in the calculation. Yes, there will be some anomalies as a result, but no more than occur when you rely on human decision-making.

Why is it that so many academics (particularly in the UK) are nervous about numbers? We could save ourselves so much time, effort and game- playing.

It is interesting to reflect again on the example of US universities. In the past 50 years, the overwhelming majority of Nobel prizes (in physics, chemistry, biology and economics) have gone to US institutions. Interestingly, out of more than 300 prizes, a third of prizewinners were born outside the US. This is evidence that American universities know how to attract the best people.

The Association of American Universities represents the top 60 US institutions - 34 public and 26 private. These universities amount to just over 1 per cent of the 4,500 degree-granting institutions in the US (this covers all public and private providers offering courses of between two and four years in length). These 60 top universities have claimed 70 per cent of all US Nobel prizes and they received 60 per cent of all federal research funding in 2011 (a total of $6 billion, or ?3.9 billion, was spent on research in these institutions). Yet they represent a tiny proportion of all US universities.

Where am I going with this?

The question is: do we have too many public universities in the UK all trying to do the same thing, including competing for the same REF money?

Similarly, the academic¡¯s life cycle should allow for different outputs at different stages. Do we really want all professors in the latter part of their careers to focus on producing top publications instead of being a wise intellectual guide for the next generation? Instead, we want some of our senior researchers to advise government, and lead our faculties and our universities. They are needed as guardians of the academy.

Andrew

I am still not sure that there is a case for a REF. The previous assessment schemes may have raised total output but I am doubtful that they have increased the amount of important work that lasts.

Amanda

You forget that public research money has to be distributed, so how are we going to ensure it goes to the very best people? Personally, I agree with Oppenheim: we should use a collection of measures that are already available and have the process moderated by outstanding scholars in the field (let¡¯s be honest, this is the same process that most hiring panels use every week). These lighter-touch assessments could be done every 8-10 years, to allow for the development of big ideas. Finally, we should incorporate a life-cycle expectation into the process.

Andrew

Maybe, but that still does not make the case for the REF in principle. These aspects would just improve the existing system.

Amanda

You will come around to my way of thinking.

In the meantime, I¡¯m knackered. Goodnight, darling.

Andrew

Goodnight, gorgeous.