“I’m blind because I see too much, so I study by a dark lamp.” This exceptionally insightful patient quote appears in the introduction to Dr Gwen Adshead’s collection of 11 patient stories, and it sets the scene for a captivating journey through the corridors of Broadmoor hospital and beyond, into the prison system, the community and the consultation room.

Drawing on Adshead’s vast experience as a forensic psychotherapist, each chapter focuses on a different person. Their crimes can make uncomfortable reading, and rightly so. We meet Tony, a serial killer who decapitated his first victim; Gabriel, who stabbed a complete stranger in a north London cafe; Zahra, who enjoyed setting fire to herself (and other people) and Ian, who sexually abused his two young sons. If this book were a tabloid article, these deeds would serve only as the lurid headlines; words intended to draw us in, to play on our endless, quiet fascination with the murderous and the macabre. Perhaps this fascination stems from an increasing desensitisation, or perhaps it’s a strange sense of collective responsibility for the dark fruit of our landscape, and the headlines provide us with a checkpoint of accountability. But beyond the headlines, and within these pages, lies the true narrative of the patients and alongside them, the calming presence of Adshead herself. “Some things I will ask you to look at will be hard to unsee,” she writes. “But I know from my own experience that gaining insight … is transformative, and I will be by your side.” It is this constant and reassuring voice, a voice not hesitant to admit to its own occasional missteps and misjudgments, that makes the book so valuable and absorbing.

Self-doubt is perhaps the most essential medical qualification of all and, as a clinician, Adshead questions her own prejudice at several points during the stories, challenging her readers to do the same. The templates from which we often work – the dutiful compliance of an Asian daughter, the moral standing of a blustering, golf-playing GP, what a child sex offender looks like – all provide a rich and dangerous harvest. Even more worryingly, these prejudices can travel beyond our subconscious and into misdiagnoses, sentencing and resources. Several of Adshead’s stories, for example, illustrate society’s preference to explain away female violence as a result of trauma (despite the vast majority of the very many female victims of trauma never becoming violent) and that assumption might lead to different care plans or the absence of certain therapies being offered.. The book also recalls an occasion when a female patient became so enraged at her male therapist that she began shouting and growling, and taking chunks out of his office door with a sharp object. Adshead, a witness to this fury and “with a speed fuelled by cowardice”, leapt into a hall cupboard and locked herself inside. She questioned, afterwards, whether she would have been more likely to intervene if the patient had been male. Having gone through an identical experience with a laundry cupboard on a high dependency unit, I now question myself too. Perhaps the idea of female violence is so distasteful to us as a society that it is easier just to look away.

In contrast, images of motherhood, from the Virgin Mary to the Hindu goddess Lakshmi, are threaded throughout our consciousness, and they also appear throughout the book as the respective mothers of Adshead’s subjects are lost to them in different, equally painful, circumstances. The order in which these stories are told is far from coincidental, and towards the end of our journey we meet a patient who challenges our definition of what a good mother should or should not be. Perhaps, though, the most moving account of maternal loss is found in Gabriel’s story. Originally from Eritrea, Gabriel arrived in the UK as a young asylum-seeker, fleeing the violence and conflict of his homeland, but becoming separated from his family in the process. With his narrative, and with other chapters, Adshead is keen to convey that while people of colour make up 13% of the UK’s general population, they represent around 25% of the populations within prisons and secure hospitals, a ratio reflected within the book itself. Statistics such as these, along with the work of theorists and historians, are peppered throughout the pages. But this is not an academic text, and though there is a gentle invitation in the notes to explore these issues further, at no point did I feel I had returned to a lecture theatre. The background and context we are given throughout the stories only add to their breadth and texture, and huge credit must be given here to Adshead’s co-author, the writer and dramatist Eileen Horne, who has collaborated with Adshead in this “joint exercise in empathy”. She retains the voice and experience of the clinician, without any danger of losing the emotional investment a reader has made in the patient.

Horne also manages to weave in a selection of quotations chosen by Adshead, from The Merchant of Venice (“prick us, do we not bleed?”) to Marvel Comics (“You won’t like me when I’m angry”), which not only lightly punctuates the grief and pain within a patient’s story, but also illustrates the many bridges between psychiatry and the arts. Given the importance of language within the speciality, it is no surprise that so many psychiatrists are also writers. Adshead discusses language many times, highlighting a patient’s selection or absence of words, the use of present tense when describing a traumatic event, and the difficulties faced when the doctor and the patient are not fluent in the same language. Even if they are, the results can still be frustrating. When asked to elaborate on something, David (the blustering, golf-playing GP) repeatedly dismisses Adshead’s questions, not with “I don’t know” or “I’m not sure”, but with “I couldn’t say”. An interesting choice of phrase, which prompts a further exploration of what it is that might be silencing him.

This collection of stories joins Nathan Filer’s excellent exploration of schizophrenia (This Book Will Change Your Mind About Mental Health) and forensic psychologist Kerry Daynes’s wonderful The Dark Side of the Mind in using patient vignettes to illustrate both mental illnesses and our perception of those illnesses. The Devil You Know has a richly deserved place on that bookshelf. Adshead’s words are effortlessly readable and deeply moving. This is not just down to the patients’ stories themselves, but to Adshead’s honest and compassionate response to those stories, and her ability to write with such clarity and elegance around even the most distressing of narratives. How a therapist responds to their patient is an important part of the therapeutic process and Adshead experiences transient fear, sadness, irritation and even drowsiness during the course of her consultations, and each for a different reason.

With that therapeutic process in mind, when I had finished reading I reflected on how those stories made me feel. I have never worked in forensic psychiatry, but the patients I met through this book made me nostalgic for the wards. Places where we are taught always to sit in the chair closest to the door; where, to an observer, the only thing that distinguishes the doctor from the patient is a lanyard; where a spoon going missing from the dining room is a grave occurrence. As in life, the stories in this book do not always have the happy ending we might crave. We may feel discomfort during a shift in our perspective. We may, temporarily, absorb the pain felt by the patients, and other victims, about whom we have just read. However, the most overwhelming feeling I had on finishing this book was of hope, not only for the patients but for the readers. Over the last 12 months we have all seen too much and therefore, perhaps, become blinded. This insightful, compassionate and fascinating book will help us to move away from our blindness and misconceptions and shine a light on the stories beyond the headlines – stories that desperately need to be heard.

• The Devil You Know by Dr Gwen Adshead and Eileen Horne is published by Faber (£16.99). To support the Guardian order your copy at guardianbookshop.com. Delivery charges may apply.