Why do religious and spiritual movements wax and wane? As pews empty in the West, spirituality steps into the breach. Elsewhere, the faithful flock to organised religion when it is allied to earthly powers. At any given point in human history, argues Linda Woodhead, religion and spirituality co-exist uneasily.

Those who enjoy the spectacle of academics being wrong-footed will have been gratified by the general failure to predict the huge upsurge of religion that has taken place in many parts of the world since the Seventies. The rise of energetic forms of Islam, Hinduism, Buddhism and charismatic Christianity exposed an embarrassing theoretical nakedness.

Some people have blamed the failure on academics' longstanding preoccupation with secularity. The existence of a large and sophisticated body of theory about secularisation was of little help when it came to explaining the equally important process of sacralisation. But it is not that secularisation theory is wrong: one has only to consider the relentless decline of churchgoing since the Seventies in most parts of Europe to see how right it was in its main predictions. It is just that secularisation theory can no longer claim to tell the whole story.

First, it is helpful to make a clear distinction between religion (the sacred as primarily a social matter) and spirituality (the sacred as primarily an individual matter).

ÖйúAƬ

If we begin with religion and consider the historical record, it becomes clear that religions flourish when they are in alliance with earthly power.

Such power may be political, economic or military. Ideally, all three. The initiative for alliance may come from the religion or from the earthly power. Ideally from both, with benefits for each party.

ÖйúAƬ

Consider the spread of early Christianity. The key factor in its success was its alliance with the Roman Empire from the 4th century. Without this, its fate would probably have been similar to that of its early rival world religion, Manichaeism, which failed to sustain political patronage and was unable to consolidate its early gains.

That is not to say that the adoption of Christianity as the official religion of empire was a fluke - just as it was no fluke that Christianity was adopted as state religion by early modern European nations or that Islam has allied itself with various political formations. Successful alliance between sacred and secular power appears to depend on two main factors. First, the secular must lack something that religion can best supply (by acting as a force of national unity, legitimating the claims of a monarch, normalising social and gender inequalities and so on). Second, there must be some homology between sacred and secular (the early Christian God bears a striking resemblance to the Roman emperor, for example, and religious ethics tend to support family values when gender roles are disrupted in wider society).

This does not mean that religion flourishes only in alliance with weak powers. Even the strongest power may "need" religion, as we see very clearly in the case of America today. As a popular bumper sticker puts it:

"God has blessed America. Now, America, bless God." The question of which comes first, the conviction of "chosenness" or the success of religion, is a classic chicken-and-egg conundrum. When imperial Britain was at the height of its power, it too experienced a period of intense Christianisation, and it too believed it was a nation chosen by God.

But there is another condition under which religion can grow. It does well not only when it is in alliance with earthly power, but also when it mobilises resistance to such power. Religion can flourish when it supports migrant identities within a hostile culture (see the film Gangs of New York ); when it defends threatened national identities (consider Ireland and Poland); and when it supports large-scale politico-cultural identities against a greater power (as in the case of postcolonial Islam facing the force of Western capitalism).

The remarkable success of Islam in recent decades is best explained by its ability to ally itself with political power in various Islamic states and to serve as a defence and rallying point against the growing power of the West, particularly the US. Islam finds itself in the sweet spot where the two most propitious conditions for religious growth coincide.

Implicit in these remarks is the idea that the sacred is an independent form of power. Being relatively free-floating, it may ally itself with all sorts of things, not only earthly powers and political regimes. Individuals may also lay hold to such power, and in doing so turn religion into spirituality.

ÖйúAƬ

Not surprisingly, religions have powerful defences against the loss of their sacred monopoly. Individuals who claim a connection with sacred power can be co-opted (usually as monks and nuns) or condemned (as mystics, magicians, witches and the demon-possessed). Recent papal denunciations of New Age spirituality indicate just how seriously the threat of spirituality is taken, and just how forcefully it is resisted. Spirituality is therefore most likely to grow when religion is not strong enough to control it. And this is most likely to happen when religion is unable to form a strong alliance with earthly power.

ÖйúAƬ

Let me illustrate this by drawing attention to a highly unusual era: Europe, 1700-1900. What was so unusual was that the Christian churches managed to attain a massive cultural hegemony, more intensive and extensive than anything achieved before or since. Alliances with worldly power were a major factor, for Christianity helped to smooth the transition to capitalist modernity by securing the precarious position of the middle classes, ameliorating the worst deprivations of industrial society, and pacifying women and workers. And under the reign of ecclesiastical power, magic, superstition and spirituality went into steep decline. Subterranean spiritual currents such as transcendentalism and theosophy were not only weak but also self-conscious in their countercultural and anti-establishment stance. The decline of magic is often attributed to the triumph of science, but I would attribute it to the triumph of confessional religion.

When religion gradually lost its grip, as its role in securing the social order and providing education and welfare diminished, spirituality began to revive. But what was reborn was not identical to what had been in place before the early modern period. That had been a form of spirituality concerned with the attainment of what may be called "objective goods" and "external prosperity". Improved technology, greater affluence and the rise of a post-materialist society now gave rise to new forms of spirituality more concerned with "subjective goods" and "inner prosperity". This is a new, subjectivised, form of magic, whose aim is to enchant not the world but the self.

In Europe then, where religion is now weak and its alliance with worldly power is minimal, we would expect to find spirituality flourishing. And so we do: from Ireland to the former Soviet Union, from Iceland to Italy. In the town of Kendal in Cumbria, where a team of us from Lancaster University recently conducted an in-depth study of religion and spirituality aimed at assessing their relative weight, we found that if the growth of spirituality and the decline of organised religion continue at their current rates, the numbers involved each week in spiritual activities willdraw level with the numbers involved each week in organised religion in about 2030.

Extending this framework, one might also predict that where religion is strong and is bolstered by alliance with earthly power, as is the case in much of the Islamic world, individualised forms of spirituality (such as some more traditional forms of Sufism) will be suffering. Again, the facts bear out the forecast.

But there is a third scenario as well: where religion is in loose alliance with worldly forms of power but is not hegemonic. In this situation, we would expect to find religion and spirituality co-existing. Yet again, the prediction is confirmed. To some extent we find this situation in America, where the churches become containers of ever-more subjective forms of Christian spirituality. The more striking and important example, however, concerns the massive upsurge of charismatic Christianity in the southern hemisphere in recent decades.

What we find in Latin America, Africa and parts of Asia is a highly deregulated form of the Christian religion, in which individuals take hold of the sacred and establish their own priesthood and their own churches.

Far from flowing in tightly regulated channels, the Spirit blows where it wills. And where it is received, it brings not only subjective healing, but also objective support and succour in making the difficult and costly transition to global capitalism. Like Islam, charismatic Christianity presents itself as postcolonial. And like Islam, though in a different way, it finds itself in a sweet spot for growth: able to draw on the resources of religion while providing many of the benefits of spirituality.

I would argue that religion is likely to flourish in alliance and/or defiance of earthly power, whereas spirituality is likely to flourish in the wake of such alliance. But I am equally keen to draw attention to the corresponding movement of secularisation that lies right at the heart of sacralisation: for when religion grows, spirituality tends to decline, and vice versa. There are certainly instances in history when the pendulum has paused at one or other end of its swing, but in the messiness of the real world it is more common to find it poised somewhere in-between.

ÖйúAƬ

Linda Woodhead is senior lecturer in Christian studies, department of religious studies, Lancaster University.