The failure of string theory to deliver on its early promise is symptomatic of a deeper problem, says Lee Smolin. Modern scientists are not taking time to step back and reflect on their work from an objective, historical perspective

Today, no one disagrees about the power and success of science. Even religious fundamentalists watch television and fly in aeroplanes. But why does science work so well? What is required to become a master of the power inherent in science? Is it tied to a particular culture or philosophy? And what is the relationship of the progress of science to the evolution of the larger society? Periods such as ours in which science evolves quickly are also periods in which society, law and the arts are all undergoing rapid change. Is this mere coincidence?

In school we are taught a simple answer to these questions: science works because of the scientific method. This is supposed to begin with an experiment, done in a laboratory, recorded according to a certain form in a laboratory notebook. (Remember when your class studied falling bodies?) The problem is that there is little evidence that Galileo - or most real scientists - worked this way. After the works of Imre Lakatos, Thomas Kuhn, Paul Feyerabend and many other philosophers and historians, virtually no one who has studied the matter thinks that the answer to why science works lies in the application of a method.

But how, then, can we judge when a theory constitutes an advance? For example, suppose a large community of experts, including many highly placed and honoured, come to believe in a theory that, unfortunately, makes no experimental predictions by which it could be tested. They argue that it is the most beautiful, most compelling theory, that it is "the new paradigm" or "the only game in town". Does this make their theory right, or at least more right than the alternatives?

ÖйúAƬ

What, furthermore, if this very compelling theory makes no predictions because it comes in an infinite number of versions, each of which predicts a different world with different particles and forces? The adherents say: "No problem, let's posit the existence of an infinite number of universes, each governed by a different version of the theory. Then the laws we observe are true just by chance, and no new predictions are possible because whatever the experiments reveal will be consistent with some version of the theory governing one of the universes."

The problem is that this makes the theory impossible to test, because to pose a real test a theory must be able to fail. The idea that a theory must in this sense be "falsifiable" was promoted by the Austrian philosopher Karl Popper, who had a long and distinguished career at the London School of Economics. But the advocates of an untestable theory can point out that some philosophers have since argued against falsifiability as a criterion for the scientific. (If I have a theory that all swans are white and I observe a red one, I don't give up the theory - I go looking for the rascal with the can of red paint.) So is there any argument against elevating a widely believed but untested theory to the status of scientific truth?

ÖйúAƬ

This is not just idle speculation. I have been describing the present state of string theory, a research programme aimed at the complete unification of all the particles and forces in nature and all the laws they follow. When many of us began working on the theory more than two decades ago we had no idea it would turn out this way. In 1984, after heroic work by the seers Michael Green and John Schwartz (now of Cambridge University and California Institute of Technology, respectively), it seemed that string theory would explain all the choices made in the formulation of the laws we then understood, and give precise predictions for myriad experiments shortly to be done.

The realisation that there was not one theory but vast numbers came quickly, after a key paper by Andrew Strominger in 1986. Yet few experts abandoned the hope for a single unification. I was among those few, and in the late 1980s I began exploring scenarios for the laws described by the many different string theories to evolve through the history of the universe. I named the space of possible string theories "the landscape", after the space of possible genes used in evolutionary theory, and - in papers and my first book, Life of the Cosmos (1997) - contemplated that our laws were the result of evolution on this landscape. I turned to natural selection because I found it would lead to precise, falsifiable predictions, in spite of the many possible theories.

For some years most string theorists told me I was wrong. Then in 2003, all of a sudden there was a sea change, and many came to agree about the landscape. Yet they often advocate "anthropic solutions", according to which the role of a theory is no longer to make predictions, because all that experiment could do is locate us among an infinitude of universes - each posited to be as real as ours.

To respond to this claim, I decided to write another book, which would make a careful argument for maintaining the traditional standards that theories must make falsifiable predictions that are confirmed by experiment before they can be considered knowledge. Indeed I was already thinking about a book that connected the success of science with the success of democratic societies in which science was born and has flourished. One day at a conference, my agent, John Brockman - who represents many of us scientists who write - made a suggestion. Why not structure the general argument around the controversies over string theory?

Thus was The Trouble with Physics born. And, as Brockman anticipated, the book has been a success - both in ordinary terms and in terms of engendering discussion. But the book has also, for me, been a kind of experiment in its own thesis. For the core idea of the book is that there is a relationship between the progress of science and the evolution of society as a whole. This connection arises from a view I propose in the book about why science works.

According to this view, science succeeds because scientists form communities of experts held together and governed by voluntary adherence to a particular system of ethics. This is based on two principles. First, when a hypothesis or question can be decided by rational argument from public evidence, then it must be regarded as so decided. To be a scientist you have to be prepared to be wrong, when the weight of evidence goes against your theories or views. Second, when rational argument from public evidence is insufficient to decide a question, the community must encourage a diversity of views and approaches to that question to flourish.

How does one then distinguish science from other endeavours claiming to be science, such as Marxism or creationism? These two principles give sufficient grounds, because they imply that to join a community of scientists a person must do two things. First, they must subscribe to and follow these ethical principles. Second, they must be trained in a craft of experimental procedure or theoretical argument that is necessary to make a convincing case to other experts from public evidence.

Applied to the case of string theory, these principles suggest that we should hedge our bets, even while we continue to investigate where the theory leads. But there are other approaches to unification and quantum gravity under development and, in recent years, the most impressive successes have come from these other approaches. The principles I have enunciated then require that the full diversity of approaches proposed by experts be supported and encouraged.

ÖйúAƬ

So it is not string theory per se but what might be called string theory fundamentalism that is under attack. To build a consensus around the view that any untested idea in science is the right one is, I argue, as dangerous for the health of science as fundamentalism of a religious or political nature is bad for the health of a democratic society.

My book was only one of several recent books that criticised string theory; others were by Lawrence Krauss, Roger Penrose and Peter Woit. What was the response of string theorists to our critiques? A number have responded professionally with arguments that acknowledge the arguments in our books. These include Philip Candelas, Brian Greene and Joe Polchinski. Meanwhile, the vast majority, including many of the intellectual leaders of the discipline, have chosen to remain silent.

However, the most visible responses have greatly disappointed my hopes to have a high-quality, critical and open-minded discussion of the issues our books raise. Many have chosen to see the publication of our books as a public relations problem and have made false claims about what they contain. A number have responded by attacking us rather than our arguments. In several debates, seminars and online postings, I and other perceived "critics" of string theory have been called "crackpots", "failed scientists" and worse. Most telling, it turned out that some of these colleagues are not embarrassed to admit they did not read the books they have spent so much time criticising.

This has raised a puzzle of its own. How could it be that highly accomplished scientists, professors in the best research universities, see no problem with publicly critiquing a book they have not read? This puzzled me greatly, but as the debates over my book raged, I have begun to understand that something deeper was at stake than the fate of string theory versus its rivals. The fact that good scientists were unembarrassed to attack books they have not read has implications for the question of the extent to which science is autonomous and independent from developments in the rest of culture, or an integral part of them.

The argument that science is autonomous stems from the ideal that scientific truth be objective and testable. Hence, to contribute to science, all a scientist needs to do is learn a craft and subscribe to the ethics of the scientific community. This universality is a great strength of science, and indeed the laboratories and institutes of many countries are as diverse in terms of national origin and culture as any workplace on earth.

But universality and independence do not necessarily mean that scientists are autonomous. There is a technical scientific literature, consisting mostly of papers describing results of calculations and experiments, which are published in specialised journals and posted on specialised websites. The question at issue is: what is the relationship between the technical progress of science, as reported in those papers, and the larger progress of human culture?

ÖйúAƬ

There is, I suspect, a deep disagreement between those who think that the specialised scientific literature is autonomous and those who think of it as a part of human progress, with myriad relations to other developments. One way to determine the difference is to ask whether it is necessary to be conversant with the history and philosophy of one's scientific field to do one's best work. And is it helpful for a scientist to be conversant with the issues that contemporary artists or architects argue about when they get together?

If you think that the logic of scientific progress is found entirely within the technical literature, then nothing else is necessary. This is certainly the dominant view. The bulk of scientists are thus, sad to say, ignorant of the real history of their field, and many have never read a book or paper in their speciality written earlier than ten years before they entered graduate school.

On the other hand, if you think that the scientific literature is part of the broad front of culture, then you will believe that what happens in the scientific literature cannot be understood in any depth without an education in the history of one's discipline and the writings of philosophers who have struggled with the elemental questions behind the concepts you use. The minority of scientists who feel this way understand their subject in an entirely different way from their historically untutored colleagues.

To come now to the point, this culture clash within science is, I am beginning to suspect, responsible for much of the underlying disagreement about string theory. Indeed, I would posit that this is a continuation of a long-standing fission in the community of theoretical physicists, which goes back to the 920s - between those who see themselves as elementary particle physicists, in the tradition of Richard Feynman, Julian Schwinger and Steven Weinberg, and those who see themselves as "relativists" in the tradition of Albert Einstein and Roger Penrose.

The style of the relativists, which has been inherited by many who work on the problem of quantum gravity outside of string theory, may be called philosophical or foundational. Its practitioners are motivated by deep philosophical problems about the nature of space, time and matter. This was the style of Einstein, Niels Bohr, Erwin Schrodinger and the other inventors of quantum theory, and it dominated physics as it was done in Europe in the first decades of the 20th century.

However, this style was supplanted in the 1940s by a much more pragmatic and anti-foundational style, led by a new generation who were tired of the (by then) old debates about the meaning of quantum mechanics. They wanted to accept quantum mechanics as given and use it to understand a broad range of phenomena from stars to semiconductors. The triumph of this pragmatic style was, among many others, the standard model of particle physics, put in final form in 1973. String theory, one of several attempts to go beyond that standard model, has been so far developed mainly within the pragmatic anti foundational style.

One of the main conclusions of my book is that to find the next great step in unification, we need a return to the older foundational and philosophical style of research. The pragmatic style is ideal for rapid development within a fixed framework, but it is impotent when the task is the invention or discovery of new mathematical and conceptual frameworks.

Those of us in the relativist tradition take as paramount a principle called background independence. This holds that there is nothing in the geometry of space and time that is fixed; all is dynamic and contingent. We take this principle to be the essential truth of Einstein's general theory of relativity and the necessary starting point of any approach to quantum gravity. We also understand the principle of background independence to be an application of principles that the philosopher Gottfried Leibniz first applied to the nature of space and time in the 17th century. String theory, on the other hand, is an extension of the methodology of elementary particle physics, and so far does not incorporate this principle.

When pragmatists write books for the public, it is likely to be for a pragmatic end, to publicise the results in their field of research. There are more and more books and articles such as this in science magazines. There are also a growing number of programmes to train young scientists to do "science publicity" as well as universities that send out press releases on the publication of scientific papers by their professors.

To the string theorists among these publicity-conscious scientists, books critiquing string theory were mainly challenges in public relations. That is why they felt it appropriate to criticise without reading them; as in a political campaign, their main goal was to discredit the perceived opponents, by slandering our reputations and questioning our motives. Furthermore, there was no reason to read the books in question because in the pragmatic tradition, reflective or philosophical books play no role in the development of science. How then are disagreements within a field to be argued? I asked this question and was told that the right forum for such discussions is private, in conversations among insiders.

However, for those of us in the more foundational and philosophical tradition, books have played a major role in the progress of science, not least by providing a venue for thinking through controversial issues. Our heroes such as Einstein, Bohr, Werner Karl Heisenberg and Schrodinger all wrote books in which they argued about the most challenging foundational scientific problems they faced. The same is true of biologists such as Charles Darwin, Richard Dawkins, Stephen Jay Gould and others. These books are written for both fellow scientists and the public. The latter are invited, as Brockman put it in one of our first conversations, "to look over our shoulders as we discuss and debate the deep issues our fields face".

Thus, rather than being a work of publicity, The Trouble with Physics was intended to be a book in this tradition of reflective books by scientists. Moreover, because of the focus on how science works, it was intended also as a contribution to the literature on the philosophy and history of science.

Books in this reflective tradition may advocate a position with regard to controversial issues, but they do not make propaganda for their point of view; they make arguments, grounded in the faith that in science it is rational argument that wins the day. They do not attempt to pull the wool over the reader's eyes by hiding the controversial and unresolved nature of the issues discussed.

Here is how to tell the difference. If you read a book about an unproven subject such as the origin of the universe, higher dimensions or string theory and the author does not emphasise that he or she is explaining one side of a controversial topic, you have in your hands a work of publicity. The author hopes you will be dazzled. If the author emphasises that the field does not yet have a consensus and that his or her view is only one among several, you are holding a work in the tradition of reflection on science. The author hopes that you will think.

There is another reason to write such books. They stimulate thought, not just in the audience, but in the author. With each of my three books I thought through a knotty issue that was confounding progress in my technical work; with the clarity achieved I went on to do technical work that I would not have thought to do otherwise.

This reflective scientific literature not only looks back in time, it looks outwards, beyond the speciality in question. When we write such books, we envisage the reader as an independent thinker who needs to be convinced by argument. These people may be laypeople, but they want the real thing, the real issues and arguments, and not publicity and propaganda. The very act of writing such a book is then an expression of faith in the abilities of human beings to resolve disagreements by rational argument from public evidence, an ability that is the foundation of the successes of science and of the democratic societies in which science flourishes.

ÖйúAƬ

Lee Smolin is a researcher at the Perimeter Institute for Theoretical Physics and an adjunct professor of physics at the University of Waterloo in Ontario, Canada. The Trouble with Physics: The Rise of String Theory, the Fall of a Science and What Comes Next is published by Allen Lane, ?25.00.