Many articles in scholarly journals are either unread or greeted with complete indifference. So what should academics do if they want to engage readers - and perhaps even "change lives"? At least in some fields, there is always the option of the popular self-help book.

It sounds like a good idea. It is only a small step, in theory, from a statement in a journal article (complete with footnotes, caveats and detailed statistical analysis) that certain kinds of parental behaviour tend to correlate positively with more confident children to the kind of bold exhortation that says "If you want a happy child, do X, Y and Z!" Other academics have privileged access to the wisdom of the ages, through studying the great writers and thinkers of antiquity or the Renaissance. So what happens when they take us by the hand and try to tell us how to be better or happier, offering guidance on getting on with our partners, enjoying our work, coping with our bereavements or getting "quality time" for ourselves?

Take the biggest prize of all, personal happiness. Lord Richard Layard, founder-director of the London School of Economics Centre for Economic Performance - and former government "happiness tsar" - is a man on a mission. In his book Happiness: Lessons from a New Science (2005), he brings together vast quantities of research on why we don't feel better and what we can do about it. But despite the 25 pages of references, it is aimed firmly at a popular market.

Layard praises "the self-help movement, especially when it aims at the inner self". He is largely concerned with broad social factors, although many have obvious implications for how individuals should behave (family breakdown and too much television, he argues, tend to make us unhappy). But he concludes with a few specific pointers: "Better to seek the beauty within than to have an affair ... The true pilgrim fights the evils in the world out there and cultivates the spirit within. The secret is compassion towards oneself and others ... ".

ÖйúAƬ

Not everyone is impressed by such "guidebooks to happiness". As the psychoanalyst Adam Phillips put it in an interview with The Guardian: "A culture that is obsessed with happiness must really be in despair, mustn't it? ... It's become a preoccupation because there's so much unhappiness. The idea that if you just reiterate the word enough and we'll all cheer up is preposterous."

There are a number of other reasons why academics are wary of offering overt guidance. The main qualities needed by writers of self-help books - empathy, worldliness, an ability to cut to the chase - are neither particularly associated with academics nor encouraged by the structures of prestige and career development in universities. And then there are the ways that authors establish their authority. Scholars underwrite their views on macroeconomic policy or Chinese history by referring to their awards, prominent positions within faculties or PhDs from elite institutions. Self-help gurus may also mention their professorships in psychology but, along with the gravitas, they also need to get across that they understand people's everyday problems. Hence the poignant, sometimes embarrassing confessions ("I always fell in love with losers until I found a way to break the pattern"). Is it possible to combine such a tone with the traditional academic virtues?

ÖйúAƬ

One solution is simply to split oneself in two and to make the time to write both academic and more popular books. Cary Cooper, professor of organisational psychology and health and pro vice-chancellor for external relations at Lancaster University, is a well-known writer and media commentator on issues such as stress, work-life balance, women in management - and, perhaps appropriately, time management. "Research is still core to what I do," he says. Along with journal articles, he averages one authored and one edited academic book a year, although their scholarly focus does not preclude implicit lessons for organisational policymakers.

But he also produces an occasional book offering explicit guidance to individuals. The most recent, co-written with Theo Theobald, is Detox Your Desk: Declutter Your Life and Mind (2007). While Cooper and Theobald distance themselves from the extravagance of many self-help books (in which "anyone can become President"), they nonetheless make bold claims for their "10-day detox programme" - namely, that it will help readers "find some time for yourself again, time to do things you want, which we'd guess isn't reading 'self-help'".

Even if such books sell far, far better than academic works, they don't count towards the research assessment exercise, although Cooper suspects that other factors are pushing more and more academics towards popularisation.

"Most universities are being told by research councils that their research has to be applied and relevant," Cooper says. "There are quite a lot of academics who can't write popular books (and some who can't write any sort of book). But I think we will see more academics writing across the board, as there is more pressure to produce work that makes a contribution. Now even grant applications ask about promotional plans and 'reaching out'. My fellow academics would definitely be sniffy if I wrote only very popular books and didn't do proper science, but there's no reason why one can't do both - like a classical musician who plays jazz at weekends." And for those with Cooper's energy and flair as a writer, one can say only: why not?

A more unusual kind of self-help book is Laurie Maguire's Where There's a Will There's a Way (2007). The author is a Shakespeare scholar and a fellow in English at Magdalen College, Oxford. When she went though a personal crisis in 1999, she read her way through "the entire self-help section of the local bookstore. And that's when I realised that I had read it all before: in Shakespeare." The 16th century "saw the beginning of self-help literature" in the work of writers such as Machiavelli and Castiglione. As well as being a cultural icon, she suggests, Shakespeare too was "a self-help guru" and "life coach".

Her book tries to substantiate this claim. Some of it is an implicit attack on current fashions in literature departments, where professional standards require that academics discussing Shakespeare "talk about epistemology and representation and semiotics and differance and liminality and cultural positions". While this can be interesting, it can also be at the expense of exploring some of the other things actors and theatregoers tend to think his plays are about - namely (to cite chapter titles from Maguire's book), anger, forgiveness, jealousy, loss and unrequited love.

Maguire is an infectious Shakespeare enthusiast. But she also wants to use him to improve our lives. As in many self-help books, she tries to establish her credentials by telling us about her own experiences. Of three flawed heroes of Shakespeare plays, she writes: "Troilus is a narcissist; Angelo is a hypocrite; Bertram is immature. I've dated them all. In fact, I had one brief relationship where my partner was all three of the above." Her friends, she says, refused to refer to him by name and called him simply "the Jerk". In the throes of another unhappy love affair with a bad-tempered man, she once wrote "a reproachful article about angry male characters" in Shakespeare's Troilus and Cressida. A journal rejected it as "strangely accusatory". It was, she now admits, "do-it-yourself therapy, not academic criticism".

So what are the life lessons Maguire wants us to learn from Shakespeare? He teaches us, for example, to "tell your family you love them. And he has a back-up plan lest the declaration of love comes too late: forgive them. Forgiveness does not undo damage, but it frees you from being disempowered by it." This may be perfectly sensible advice, and we can all think of people who could learn from it. But does one really need Shakespeare to tell us this? And doesn't it amount to a pretty banal "message" (or soundbite) to get from a play such as King Lear?

ÖйúAƬ

One of the problems with academics writing in this mode is that some of the most powerful attacks on the self-help industry have also come from academics. Maguire praises in passing a number of bestsellers aimed at the confused and broken-hearted. It is precisely these books that are the focus of a very entertaining polemic by Deborah Cameron, Rupert Murdoch professor of language and communication at the University of Oxford, entitled The Myth of Mars and Venus (2007). Although she is also critical of academics who have written on sex differences and communication styles, such as Simon Baron-Cohen, Steven Pinker and Deborah Tannen, their work is obviously rooted in evidence and open to challenge. Popular self-help books such as John Gray's Men Are from Mars, Women Are from Venus (1992), by contrast, she says, "are not accountable (and) use anecdotes instead of evidence".

Certain oft-repeated claims about how men and women "speak different languages" - notably that "women talk more than men", that "women's talk is co-operative and men's is competitive" and that "men and women systematically misunderstand each other" - just don't fit the facts revealed by serious research, Cameron says. And her criticisms of the self-help genre are based firmly on "traditional academic values: consider all the evidence, don't generalise, acknowledge complexity".

But the problem goes far deeper, claims Cameron. Sex education and rape prevention programmes often advise women rejecting unwanted sexual advances to "Just say no". This suggestion builds on the "Venus and Mars" premise of mutual incomprehension between the sexes, that "men have trouble understanding any refusal which is not maximally direct". Yet this premise is patently absurd. In every other area of life, nobody ever turns someone else down by "just saying no", because it comes over as unacceptably rude and aggressive, as men know just as well as women. In a situation that is already tense and potentially violent, an unvarnished "no" is likely to be ineffective, if not counterproductive. And rapists have been given an additional line of defence whereby women accusing them are now expected "not only to prove that they did not consent to sex, but also that they refused in a manner sufficiently direct to preclude misunderstanding".

Where self-help gurus don't just pluck ideas out of thin air, Cameron argues, they rely not on research but on "the normative ideals of communication" promoted by the therapeutic professions. The results are often disastrous: "Self-help and therapeutic regimes deform people's unconscious but often effective ways of dealing with difficult things."

Because the term is often used pejoratively, few writers (and even fewer academics) refer to their own work as "self-help books", even in cases where many other people might.

Adrian Furnham, professor of psychology at University College London, has been cited as the second most productive psychologist in the world on two separate occasions. He has had weekly columns in The Daily Telegraph, Financial Times and The Sunday Times, and has written 11 popular books as well as 44 academic books, mainly in the fields of personality and occupational psychology.

ÖйúAƬ

Within the academy, he explains, there are few incentives to write in a popular style (except in the rare cases where a book makes "real money"). Indeed, Furnham says, there are positive disincentives. "In some fields, the less applied your work, the higher your status. The 'softer' areas are less prestigious yet more popular, and hence one gets more invitations from publishers to write about them. You should try to earn your spurs before you can break the rules; do the serious stuff to establish your reputation before the 'naughty' popular stuff. And you need to be 'bilingual' to write popular books: the style is completely different."

Furnham specifically describes his books - including The Incompetent Manager and Dim Sum for Managers - as "popular" rather than "self-help". The self-help genre, he argues, tends to split into two categories: the naive "You can all do/have it" mode and the "You are a victim" type. He shares the general academic view that both tend to be "thin and trivial" and based on little evidence, and he regards the latter as positively dangerous.

His own books, by contrast, are rooted in research and offer something rather different. "I don't tell people what to do, but I try to help them understand the issues better," he says. "I put technical ideas into everyday language and bring academic scepticism to bear on the claims made for processes such as brainstorming, so I make clear what things are likely to work and why mythologies have grown up around others. And I hope to present evidence to general readers which will make them more sceptical of snake-oil salesmen. When I get occasional hate mail, it tends to be from people whose pretensions I've undermined."

Because the self-help genre can prove so lucrative, publishers are often tempted to "package" their books that way even when the content is more complex. Take The Sister Knot (2007) by Terri Apter, senior tutor at Newnham College, Cambridge. Against a yellow background, one young woman smiles winningly at us while another looks on sideways with an envious scowl. The text is lively and accessible and includes extracts from in-depth interviews that wouldn't be out of place in a women's magazine. Annette, apparently in her mid-thirties, recalls of her older sister: "I would take her boyfriends. Every one I could. I never thought about whether I liked them. I liked them simply because they liked her. At that time my entire sense of what I was worth was related to what my sister was worth ... even now Dorri can't believe I won't steal men from her. Yes - even though she knows I'm gay!"

Although it is "based on new research", Apter says, her book is "written for both general and professional audiences. Writing for a general audience is a challenging discipline and sets somewhat different standards with different rules. But the general audience I have in mind when I write is none the less exacting in coherence of argument and convincing evidence ... A good writer does not want to cheat a general audience of sound research. Moreover, a writer who is also an academic needs to face colleagues and defend the work he/she does. I have heard one academic who wrote a very popular book about gender differences defend misleading statements by saying they were meant to be 'provocative'. For most research-led writers, this would not justify overstatement or simplification."

Despite the popular style, Apter offers neither the glib advice nor the cheap solutions one often finds in self-help books. She wants us, instead, to come to terms with some rather bleak truths: that it is rare to "overcome all anxiety about our place in the world" and that "throughout life ... the person who has been like us" but "takes a step forward or makes a sudden gain" may "trigger the poisonous anxiety of envy". And she also wants "to revitalise psychological assumptions about who we are and how we interact with others, and to have this resonate with other people who are trying to make sense of their interpersonal worlds. This focus means that my work might be in the slipstream of mainstream academia. I can live with that."

Some may feel that the tension between different aims and markets can produce a rather jarring effect when a writer attempts to reach both constituencies in one book. Yet there is, of course, no theoretical reason why the serious thinking that goes on within universities should be cut off completely from the practical wisdom that self-help books claim, justifiably or not, to offer. It is precisely in an attempt to bridge this gap that Acumen Publishing has commissioned The Art of Living, a series of books edited by Mark Vernon, honorary research fellow at Birkbeck, University of London. The first eight volumes - Sport, Wellbeing, Illness, Pets, Clothes, Deception, Fame and Hunger - are being published in September.

These titles, Vernon explains, are "personal reflective essays largely written by philosophers or academics". Most of the big-name philosophers of the past did not operate within formal career structures but "adopted philosophy as a way of life". And today, he says, there are signs that "philosophy is turning outwards again, to explore how we can live more fully".

Writing for such a series often requires authors to rethink the communication styles that have become part of their professional identity. "Academics train themselves to write in particular ways," notes Vernon, "as when philosophers try to adopt a scientific ideal and to write 'from nowhere' - but these books all have a voice. The authors mustn't hide behind technical language and must try to touch people and take risks though self-revelation. All have some serious engagement with major philosophers of the past. They are no less rigorous than standard works of academic philosophy, but are looking from the outside in, with no argument for argument's sake."

A powerful example is the book written for the series by Havi Carel, senior lecturer in philosophy at the University of the West of England. She has long been interested in how a sense of our own mortality has a fundamental impact on the way we live our lives. The topic acquired a tragic immediacy when out of the blue, at 35, she was diagnosed with a rare and incurable disease, lymphangioleiomyomatosis, for which the prognosis was ten years from the onset of symptoms.

Even now, she writes in Illness: The Cry of the Flesh, she teaches a philosophical course on death where she calmly considers "questions and intellectual puzzles while inside me a multiple-injury train crash has taken place". But while "philosophy, at least in the way it is practised in Western universities, is seen as objective and impersonal", it has also been her "strongest ally in coming to terms with life as an ill person".

So how has philosophy helped her cope with the terrible blow that has befallen her? "I consider myself extremely lucky," she says, "that I could live the life of the mind even in a wheelchair, lucky to have access to immense intellectual resources. Epicurus and Aristotle are useful in defining what it means to lead a good life. I was very impressed by how helpful Epicurus can be in such situations, by delimiting an area of the present."

While Carel acknowledges "the enormous weight of emotional labour that healthcare professionals undertake", their interactions with patients are notable for "denial, over-technical discussion and not dealing with the emotional angle".

Friends can be equally uncomfortable and evasive, which "leads to a loss of words and an inability to communicate about the most important things in our lives. The implicit demand on ill people not to talk about what is happening to them is extremely distressing." Phenomenological philosophers such as Martin Heidegger and Maurice Merleau-Ponty, by contrast, provided resources to focus on the lived reality of illness, which can again help one define - and work towards - "a good life despite ill health".

Carel's attempt "to bring the experience of illness to light" could only have been written out of a combination of personal disaster and philosophical insight. The result could hardly be less like the traditional jaunty self-help book, with zippy bullet points, losers finding love and lives in the gutter looking up to the stars.

But it would be hard for a healthcare professional or anyone who has ever been callous or insensitive to an ill friend or relative (which means just about everyone) to read Carel's book without sense of shame and a desire to do better. And, hopefully, it can also help relieve at least some of the anguished confusion of the seriously ill. If this is what academic self-help can do, long may there be more books like it.

ÖйúAƬ