When people find out I’m a journalist, they typically tell me they used to write for their school newspaper or that their child wants to be a journalist, or that their cousin is a blogger. Or they might say they loved a film about a newsroom, but can’t remember the name. They might try to look up the film on their phones and, while they are at it, will show me a cat video trending on Twitter.

It’s rare that people don’t interrupt and shift conversation to themselves. Bad listeners are not necessarily bad or boorish people. You’re likely to have a dear friend, family member, or maybe a romantic partner who is a bad listener. Perhaps you, yourself, are not the best listener. And you could be forgiven. In many ways, we have been conditioned not to listen.

Schools and universities have courses in debate, rhetoric and elocution, but rarely classes that teach listening. You can get a doctorate in speech communication and join Toastmasters International to perfect your speaking skills, but who strives for excellence in listening? Social media has given everyone a virtual megaphone to broadcast every thought, as well as the means to filter out contrary views. Online and in person, it’s all about defining yourself and shaping the narrative. The very image of success and power today is someone prowling around a stage with a microphone.

The cacophony of modern life discourages listening. Noise levels in restaurants make diners strain to hear one another. Offices are open design so every keyboard click, telephone call and post-lunch belch make for a constant racket. Traffic noise on city streets, piped music in shops and the bean grinder at your local coffeehouse exceed the volume of normal conversation by as much as 30 decibels and can even cause hearing loss. All this when listening is arguably more valuable than speaking. Wars have been fought, fortunes lost and friendships wrecked for lack of listening. The American president Calvin Coolidge said: “No man ever listened himself out of a job.” It is by listening that we connect, comprehend, co-operate, empathise and develop as human beings. It’s fundamental to any meaningful relationship – personal, professional or political.

I’m a listener by profession but I’m also a listener by nature. Perhaps it comes from growing up in Texas, a state known for its raconteurs. I was surrounded by colourful relatives and neighbours who knew how to tell a good tale – some taller than others. I learned early on that listening to the same story told by multiple sources got you closer to the truth.

But today people resist listening to one another. They find face-to-face encounters trying and phone calls intrusive, preferring text and wordless emojis. Many people refuse to listen to opposing views, shouting down or walking out on speakers who challenge their thinking. If people are listening to anything, it’s probably through earbuds, safe inside their own curated sound bubbles.

In the course of researching my book, I asked people on five continents, “Who listens to you?” and the response was usually a long, awkward pause. Even those who were married and claimed vast networks of friends struggled to come up with someone who they felt truly listened to and understood them. It’s fuelling what public health officials are calling a worldwide epidemic of loneliness, which increases the risk of premature death as much as obesity and alcoholism combined. Indeed, studies link loneliness with heart disease, stroke, dementia and poor immune function.

Research indicates great swaths of the populations in the United States and the United Kingdom do not have meaningful in-person social interactions, such as an extended conversation with a friend, on a daily basis. Many report feeling isolated and left out even in the company of other people. Indeed, the UK was moved to appoint a “minister for loneliness” to help its estimated nine million citizens who often or always feel lonely.

To combat loneliness people are told to “Get out there!” Join a club, take up a sport, volunteer, invite people to dinner, get involved at church, or just start a conversation. But that’s not practical advice when people feel lonely in the presence of others. How do you connect with people once you’re “out there” and “face-to-face”? Truly listening to someone is a skill many seem to have forgotten or perhaps never learned in the first place.

After a couple of years studying the neuroscience, psychology and sociology of listening, as well as consulting some of the best listeners out there (including a CIA agent, focus group moderator, radio producer, priest, bartender and top furniture salesmen), I learned that listening goes beyond just hearing what people say. It’s also paying attention to how they say it and what they do while they are saying it, in what context, and how what they say resonates within you.

Listening is not about simply holding your peace while someone else holds forth. Quite the opposite. A lot of listening has to do with how you respond – the degree to which you elicit clear expression of another’s thoughts and, in the process, crystallise your own. It starts with an openness and willingness to truly follow another person’s story without presumption or getting sidetracked by what’s going on in your own head. This can be a problem for smart people whose galloping thoughts may race ahead of the speaker’s words, often in the wrong direction. People with higher IQs also tend to be more neurotic and self-conscious, which means worry and anxiety may hijack their attention.

Good listeners ask good questions. One of the most valuable lessons I learned as a journalist is that everyone is interesting if you ask the right questions. If someone seems dull or uninteresting, it’s on you. Good questions don’t have a hidden agenda of fixing, saving, advising, convincing or correcting. They don’t begin with “Don’t you think…?” or “Wouldn’t you agree…?” and they definitely don’t end with “Right?” The idea is to explore the speaker’s point of view, not to sway it.

Also avoid asking appraising questions like “What do you do for a living?” and “What part of town do you live in?” and “Are you married?” These are not honest attempts to get to know people so much as rank them in the social hierarchy. It makes people reflexively defensive and is likely to reduce the conversation to a self-promoting elevator pitch or CV recitation. The kinds of conversations that make you long for the company of your dog.

Instead, ask about their interests. Try to find out what excites or annoys them – their simple pleasures and what keeps them up at night. Ask expansive questions such as, “What’s the best gift you ever received?” and “If you could live anywhere in the world, where would you live?” Listening to people in this way is also how to bridge differences and find common ground. Once you find out someone has struggled with an illness, cannot resist chocolate, hums Queen’s Bohemian Rhapsody when nervous, or has a room in their house dedicated to their yo-yo collection, it’s hard to reduce them to a political position or ideological stance.

You might not agree with them, but you gain understanding about their background and influences, which is essential to reaching compromise, or, at the very least, maintaining a peaceful coexistence. Moreover, listening to others makes it more likely that they will listen to you. This is in part because it’s human nature to return courtesies, but also because you learn people’s values and motivations. With this you’ll be better able to craft a message that resonates.

Listening is a skill and, like any skill, it degrades if you don’t do it enough. It takes awareness, motivation and practice. While some may have more natural ability and others may have to try harder, everyone will benefit from the effort. The more people you listen to, the more aspects of humanity you will recognise and the better your judgements, instincts and intuitions. We are, each of us, what we attend to in life. To listen poorly, selectively, or not at all, is to limit your understanding of the world and deprive yourself of becoming the best you can be.

You’re Not Listening by Kate Murphy is published by Harvill Secker, Vintage at £16.99. Buy it for £13.59 at guardianbookshop.com