International performance comparisons are valued by university staff around the world, but current ranking systems are sometimes seen as being based on insufficient information, employing dubious data and resulting in "insidious effects".

This is the conclusion of a survey for research data company Thomson Reuters, which is to supply all the data and analysis for Times ÖйúAƬ's world university rankings for 2010 and beyond.

The report, released on 11 February, says there is "little doubt that league tables matter".

"If well developed, they can be informative to students and their mentors, and they matter hugely to those who run universities.

ÖйúAƬ

"But they can hide as much as they show, because complex organisations span cultural boundaries and support multiple missions. No single indicator can capture that."

The survey by Ipsos Insight Corporation records the views of 350 academics and administrators from more than 30 countries.

ÖйúAƬ

About 40 per cent of respondents said that rankings systems were "extremely or very useful" while a further 45 per cent said they were "somewhat useful".

"However, the overriding feeling was that a need existed to use more information, not only on research, but also on broader institutional characteristics," the report says.

The survey found that the data indicators and methodology currently used "were perceived unfavourably" and that in North America and Europe "there was widespread concern about data quality".

Another major concern was that rankings could have "insidious effects", by changing institutions' strategies, "not to become more effective but to perform well against arbitrary criteria".

The report, by Jonathan Adams, head of research evaluation at Thomson Reuters, says: "This is of great concern and warns against any reliance on indicators that could be manipulated without creating a real underlying improvement."

ÖйúAƬ

THE is working with Thomson Reuters to develop a more rigorous and transparent version of its annual world university rankings, and the survey will inform the development of the new methodology.

The magazine is no longer working with its former rankings partner, the careers firm QS.

Respondents to the survey strongly favoured the use of research metrics such as data on research publications and citations in rankings, something that THE is embracing.

ÖйúAƬ

Ann Mroz, editor of THE, said: "By working directly on the rankings with the world's leading research metrics company, we are able to employ research data in a much richer and deeper way."

The survey also endorsed the use of reputation surveys in rankings: 79 per cent of respondents said that "external perception among peer researchers" was a "must have" or "nice to have" indicator. THE has confirmed that it will use a reputation survey in its 2010 rankings.

Dr Adams said that the survey's sample size would be about five to ten times larger than that of previous exercises by QS. And he thanked THE's editorial board of ÖйúAƬ experts for valuable insight into the content of the survey questionnaire.

phil.baty@tsleducation.com

?

ÖйúAƬ