There are precious few biological tests for mental disorders, so diagnosis still rests on the observation of symptoms. Susannah Cahalan’s first book, Brain on Fire, described her first-hand experience of how catastrophically wayward such diagnosis can be.

Cahalan was working as a reporter at the New York Post in 2009 when, aged 24, she developed what appeared to be a kind of paranoid schizophrenia. She believed bedbugs were invading her apartment, that her father had tried to abduct and kill his second wife, she heard voices, spoke gibberish, was unable to sleep and descended into catatonia. It was only the persistence of her parents, and an extra series of tests, that discovered an extremely rare autoimmune disease that was attacking her brain. The diagnosis saved her from the psychiatric ward, where, left unchecked, her illness would have left her cognitive functions irreparably damaged.

That survivor’s story led her into the history of mental illness and its treatment and, inevitably, to one of the most infamous of experiments, the 1973 study On Being Sane in Unsane Places, conducted by a Stanford psychologist, David Rosenhan. The study, published not long before the release of the film One Flew Over the Cuckoo’s Nest, involved a series of eight healthy patients presenting themselves at mental hospitals with made-up symptoms. In each case, in Rosenhan’s account, the patients fooled the psychiatrists and were admitted to secure wards. The publication of these findings caused a huge stir in the world of psychiatry, as claims of objective certainty in diagnosis were apparently exposed as a sham, and subsequent treatment haphazard and invalid.

Given her own experiences, and her twitchy investigative reporter’s nose, Cahalan was drawn to re-examine the case and its repercussions in the practice of psychiatry. The result of that inquiry is this compulsively readable book, in which she tries to track down the eight anonymous “pseudopatients” and examine the facts of their hospitalisation. Rosenhan died in 2012, the obituaries noting his famed experiment; with access to his original field notes, Cahalan was able to piece together the untold stories of his research and test them against reality.

She found only two of the eight pretend psychotics – as well as confirming that Rosenhan himself was one of them. Her efforts are so indefatigable that you are inclined to believe her hunch that the other five case histories may well have simply been made up. Even the detailed experience of those patients she was able to examine does not tally with Rosenhan’s claims. The evidence of one of them, Harry Lando, was excluded from the final report because he confessed a positive experience of the treatment. “HE LIKES IT,” Rosenhan noted, with incredulity.

Despite the apparent shakiness of methodology, noted by opponents at the time, and confirmed here, Rosenhan’s seductive findings had an enormous impact on the practice of psychiatry across the world. Within a decade, the antipsychiatry movement, fuelled by fears of asylums as a form of social control going back centuries, had helped to close 50% of American institutions. Faith in the authority of diagnosis, Cahalan argues – for better and for worse – never quite recovered.

“Whenever the ratio of what is known to what needs to be known approaches zero,” Rosenhan once wrote, “we tend to invent ‘knowledge’ and assume that we understand more than we actually do. We seem unable to acknowledge that we simply don’t know.”

The professor’s instincts were to suggest that we’d had enough of experts, but, if Cahalan’s investigation is correct, he apparently stopped short of applying that wisdom to his own scientific work.

Cahalan, having unearthed these damning caveats, is not as damning as she might be. “Rosenhan’s paper,” she argues, “as exaggerated and even dishonest as it was, touched on truth as it danced around it.” That truth was the exposure of “the role of context in medicine”, the importance of diagnostic doubt as well as faith. Is it possible, as Chief Bromden observed in One Flew Over the Cuckoo’s Nest, that Rosenhan’s findings represented “the truth even if it didn’t happen”? Cahalan is honest enough as a writer to leave that question hanging, having presented important and spirited cases both for the prosecution and the defence.

• The Great Pretender by Susannah Cahalan is published by Canongate (£16.99). To order a copy go to guardianbookshop.com. Free UK p&p over £15