I felt like I was learning how to participate in an asylum," says American sociologist Sudhir Venkatesh. He is referring to his life on the mean streets of Chicago, among the hustlers, the prostitutes and the drug dealers, which he has vividly reconstructed in his new book, Gang Leader for a Day. But he also means his parallel life at the University of Chicago.

In 1989, Venkatesh started hanging out with the Black Kings, befriended their charismatic leader J.T. and acquired a deep first-hand understanding of urban poverty. Eventually, as readers of Steven D. Levitt and Stephen J. Dubner's Freakonomics may remember, he managed to get hold of the gang's "accounts" and demonstrate how it was much like many other businesses, with a hierarchy stretching from the henchmen on street corners barely earning a living to a few millionaires at the top of the pile.

Yet all the time he was gathering data from J.T. and his associates, Venkatesh was also using his research to take his first steps up the academic ladder. He is currently professor of sociology and African-American studies at Columbia University.

"Which brand of lunacy was I supposed to be learning?" he asks now. "In the academy, you had a set of rules based on e-mails and very impersonal sanctions. On the streets, everything was immediate. I felt schizophrenic throughout that period. I don't know if I've ever reconciled those opposing worlds, but I think I've managed to find a creative tension between them, so that I'm equally at home drinking a $2 bottle of wine in a housing development or an '82 Bordeaux in some of the other places where I hang out."

ÖйúAƬ

Venkatesh was born in India but soon moved to the middle-class suburbs of California, which left him with a sense of "growing up in American society but not feeling I was really a part of it". It may be because of this, he suggests, that his research has gravitated towards marginal figures at both ends of the economic spectrum, from sex workers to the children of ultra-rich "blue bloods". He studied maths before turning to sociology, with the intention of exploring Indian attitudes to mental illness.

Yet this too proved a false start. "I gave up mathematics and cognitive science in graduate school," he explains, "because their abstract language was not my language. Then I was going to study India, because that was what I was supposed to do, but it also felt distant." It was only when he began to immerse himself in the lives of the black urban poor, within walking distance of the University of Chicago, that Venkatesh at last found his metier.

ÖйúAƬ

His new book raises many acute questions about the methods and ethics of immersive sociological fieldwork. "If I find out that the gang plans to carry out a murder," Venkatesh once asked his supervisor, "should I tell somebody?"

Such fieldwork comes in many different forms.

Take On the Make, a study of "the hustle of urban nightlife" in Philadelphia (Chicago University Press). Author David Grazian, associate professor of sociology at the University of Pennsylvania, assures us with a straight face that his "research" involved "paying visits to over 175 downtown restaurants, cafes, taverns, nightclubs, cocktail lounges, rock and jazz venues, private social clubs, hotel bars, and late-night eateries". After asking his wife's permission, he reports, he "invited four women in their mid- to late twenties to the lounge at Tangerine for cocktails" and "an eye-opening and frank conversation" about how they would go about picking up men.

Much more genuinely eye-opening is Peter Moskos's Cop in the Hood (to be published by Princeton University Press in May). Moskos, who is now an assistant professor at the John Jay College of Criminal Justice in New York, did the research for a PhD, which forms the basis for his book, simply by joining the Baltimore police for "six months in the academy and 14 months on the street".

He admits to feelings of "empathy" towards fellow officers who, like him, would put their lives on the line for "those I didn't know and those I did know and didn't like". And he confesses that he found the terrible East District ghetto "exotic". But despite his confessedly "unscientific methods", Moskos offers a compelling account of why a uniformed police patrol "does little but temporarily disrupt public drug-dealing" - and hence why "the war on drugs" is so hopelessly self-defeating.

Although it is written from below rather than above, Gang Leader for a Day reaches many similar conclusions. It conjures up a disturbing world of vicious police corruption and drive-by shootings, where gangs organise midnight basketball games, act as law-makers as well as law-breakers and even send out petty hoodlums on voter-registration drives.

But it is also full of vivid and amusing details about Venkatesh's attempts to learn about the lives of the poor. He started by going up to people and trying to administer a traditional multiple-choice questionnaire: "How do you feel about being black and poor? Very bad/Bad/Neither good nor bad/Somewhat good/Very good." When he creates a writing group for young women, who describe their extraordinary and often shocking survival techniques, the rumour goes round that he has set up as a pimp.

ÖйúAƬ

Although many initially assumed he was Mexican or "Ay-rab", his Indian background turned out to open doors for Venkatesh: "The older generation of black Americans who had grown up in the civil rights era were incredibly curious about India, about Gandhi and non-violence. They knew more about India than I did, but it helped me enter their world."

The crucial factor, however, was his unexpected friendship with J.T., who gave him "access" and protection and even offered him a personal driver while clearly hoping Venkatesh might write his biography. He soon took to popping into J.T.'s mother's flat for meals and writes frankly about his fascination with J.T. himself. It is one of great strengths of the book that it is so open about the emotional factors that inevitably feed into sociological research.

ÖйúAƬ

"His image of himself," explains Venkatesh, "was far from how the academy viewed the poor blacks living in the housing projects. Like journalists, we saw them largely in terms of what they were lacking - they don't have jobs, they don't have families, they don't have viable institutions. I was curious how, as a member of a violent street gang, he could still see himself as someone who represented his community."

Yet if J.T. attracted the budding sociologist in Venkatesh, he also intrigued him on a more personal level: "We were close in age and similarly ambitious. I was trying to make a name for myself in the academic world, and he was doing the same on the streets. We had a certain bond, we understood each other's motivations. The attraction of being able to go so deep into a world that few other people knew about was always pushing me on - and perhaps pushing me too strongly, which is where my relationship with the other key figure in my life, my academic adviser (and poverty expert) William Julius Wilson, came into play."

When he started work, Venkatesh hoped to provide insights into the life of the poor which could help mould more effective public policy. What he soon learnt on the streets is that academic research is also a form of hustling - not for food or sex or money but for vitally important information. If he wanted to find out how many people were living in a particular apartment, for example, this was not some neutral statistic but data that could effect access to benefits or residence rights.

Yet those he spoke to soon spotted his steely determination: "They might have seen me as an intruder, as an agent of the state or all sorts of other things, but they saw me as a hustler - young, interested in a prized commodity and willing to do whatever it took, which in my case meant staying in the neighbourhood and differentiating myself from other kinds of information-seekers who came and left."

Today, after working in the field for almost 20 years, Venkatesh is doubtful about how far poverty research has actually helped the poor. He is understandably wary of quick-fix solutions and believes academics need to be more modest. "American social scientists have an inflated sense of their capacity to do good in terms of public policy," he says. "I think we'd be better off if we tried to organise our intellectual activity around enlivening public discourse, using analysis to help people better understand the world." His new book provides one of the essential tools.

Sudhir Venkatesh's Gang Leader for a Day is published by Allen Lane at ?18.99

ÖйúAƬ