The clocks have changed for winter and night falls early. I’m in a small wood near my house in west Wales. My hands and feet tingle with cold. I can hear the stream rushing, a rustle in the wet leaves. There was a full moon at the weekend, so the nights are bright. Under the trees there’s a smudge of faint bluish light, a slide of shadow. This has its own magic, but I’m looking for the dark.

I like walking with the dark and I’m fortunate to live in a place where it’s safe to do this. I’m also lucky because real darkness still exists in parts of Wales. Urban areas in particular can be stripped of the dark. London, which is awash with unnecessary illumination, is 24 times brighter than dark areas of south-west England, according to a London Assembly report. Light corridors from trunk roads and motorways are clearly visible from satellites. But on a moonless night, walking in these woods – a long way from streetlights and houses – the familiar paths, trees and contours are all hidden. If I hold out my hand, I can’t see it. I only know my feet are there because I can feel the tug of my wellies.

This kind of experience is increasingly uncommon. Darkness is being stripped at a rate unprecedented in human history – the spectacle of the Milky Way in the night sky is now invisible to 80% of the world’s population (and 99% of the population of the US). There have been some efforts to reduce light pollution: in December 2021, for example, New York City passed landmark lights-out laws, while Germany, France and Spain have reduced illumination on municipal buildings. But this goes nowhere near to repairing the damage artificial light is causing.

This matters because the dark matters. Not only does darkness offer unique physical and mental benefits to humans, it is vital to plants, mammals, birds, reptiles and insects – and not just the ones that are active at night.

The trouble is, the dark scares us. In 1897, the psychologist Stanley Hall, a friend of Sigmund Freud and Carl Jung, showed how this terror begins with the “impossible monsters” of childhood nights and stays with us throughout our lives as “the mother of fears”. More recent research discovered that fear of the dark emerges in nearly all children at about the age of two, no matter what their culture or upbringing, intensifying until they are five. This fear plays a crucial role in helping us understand our place in the world, alerting us to a state in which nothing is as it seems and nowhere feels safe. We’re often more alert to the dangers of accidents and crime at night. Historically, when many communities lived in unlit towns, the chances of falling into a stream or over a cliff, or of becoming the victim of burglary or a beating, were higher at night. It’s no coincidence that for many religions and cultures, the arrival of the new moon is a sacred moment when darkness is overcome.

Emerging from these innate fears, the dark has a long history of negative associations. Since Roman times, it has been connected to both bleak moods and treacherous actions: the Roman scholar Quintilian defined “bad intention” as “something done … at night”. For centuries, and in many languages, darkness was a common metaphor, or synonym, for death and we talk of a miserable “black mood”, of the evil of “black magic”, of the financial disaster of “Black Monday”. The correlation between long periods of darkness and depression is well established, and contributes to our understanding of seasonal affective disorder, which feeds on low light levels. When things are about to go wrong in a novel or a film, darkness closes in, playing on our natural fear response.

All this reinforces the idea of darkness as a state to be avoided or an impairment to be fixed. We do a good job of pushing it back: streetlights, headlights, house lights, computers, televisions – all flash into life as night approaches. But the ongoing battle to eradicate the dark overlooks its importance to our wellbeing.

Since the widespread adoption of electric light at the end of the 19th century, time has been distorted, allowing us to work longer hours, comfortably shop, cook and read after dark, to move about and socialise more safely. Artificial light has been a comfort and a distraction, but it’s also cast us adrift. Our body’s circadian time-keeping system, which allows us to respond to environmental changes, is naturally sensitive to the dark. As day comes to an end, these circadian rhythms prompt clocks throughout the body to take note and prepare for sleep, eliciting shifts in temperature, blood pressure and hormones. But without the signals provided by the arrival of darkness, these rhythms can falter, creating a disruption that distresses our entire body.

This is important because the systems controlled by the circadian cycle play a profound role in our physical functions, including ageing, cell proliferation, cell death, DNA repair and metabolic alteration. When circadian rhythms are confused or disturbed, this affects the body’s metabolic system, raising the risk of developing conditions such as diabetes, cancer, obesity or cardiovascular disease. Immune deficiency and high blood pressure – with the accompanying risk of stroke – have also been linked to circadian rhythm disturbance.

For millennia, our circadian rhythms have been protected by a consistent regime of day and night, light and dark. But this has changed. While research into the long-term effects of our bright nights is in its early stages, it’s becoming clear that a constant immersion in light is changing the way our bodies work. Hormones respond particularly sensitively. An absence of darkness disturbs the production of glucocorticoids from the adrenal gland, which are important in regulating stress responses: a disturbance in glucocorticoid production is associated with a significant number of mood disorders. A curtailed or eliminated dusk also means the body fails to secrete the melatonin that we need to prepare for sleep and which dictates sleep timing and quality. Deprived of darkness, our brains don’t work well.

At home, we can turn off the lights, shut down devices, black out the windows. But outside, there’s not much we can do. Planning laws rarely deal with light pollution, and those of us living in the brightest areas experience worse sleep and are more likely to report insomnia and snoring. We’ve grown so accustomed to a certain level of illumination even in the middle of the night that we might not notice it, but the light intensity on an average urban street is estimated at 5-15 lux – much less than daytime light, of course, but way beyond the 0.001 lux of a starry night and significantly more than a full moon on a clear night, which generates 0.1-0.3 lux, or up to 1 lux in tropical countries.

As with much other habitat loss, the scale and implication of these changes are often hidden. We might guess how an absence of darkness impacts nocturnal hunters: owls, bats, cockroaches. But when we look more closely, we see that the effect is more far-reaching. As the dark disappears, hatching turtles are disoriented; camouflage ceases to conceal. Reproductive cycles fail and the daily life cycle of plants is interrupted. Animals and insects stay active for longer, exhausting themselves; night pollinators like moths lose their ability to see flowers. By some estimates, more than 300 million birds – or perhaps as many as a billion – die annually worldwide because the lights from tall buildings interfere with migration; it has been shown that in the glow of city lights, birds sing less.

Take the humble sand hopper. These tiny, transparent, flea-like beasties wait for night to navigate from burial spots in the sand to the open shore, where they feed on seaweed. They need the dark. On beaches affected by nearby lights, sand hoppers become confused and disoriented. Some act as though day has burst on them unexpectedly, and scurry back to the sea where they feel safe. Some fail to venture out in the first place. In either case, these foragers go without food.

I don’t suppose many of us have great fondness for the sand hopper. But adaptations in the behaviour of one creature signal more widespread change. The same light pollution that confuses a sand hopper threatens an entire ecosystem: the many species of animals and plants living on beaches and rocky shores, and even on the seabed. Almost three-quarters of sea floors close to affected sand hoppers are damaged by the leak of light from the land. Since 75% of the world’s megacities are located in coastal regions, and coastal populations are forecast to double by 2060, it’s easy to see how much of the world’s marine habitat is at risk from the simple fact that the dark is disappearing.

My night walks are about learning to understand and embrace darkness as a distinctive habitat while I still can. I’m not alone. “Dark-sky tourism” is proving increasingly popular: the UK has seven Dark Sky reserves and two International Dark Sky Parks raising awareness of the wonder and strangeness of darkness. When we lose the dark, we lose a sensory richness. In Under Milk Wood, set a few miles down the coast from where I live, Dylan Thomas celebrated “the sloeblack, slow, black, crowblack” dark of a moonless night, a darkness that is felt as much as seen. For those of us who can usually see, the blindness of the dark allows us to put aside the tyranny of sight to experience our world differently. Think about walking by a freshly cut field at night. Unable to see the field, we’re thrown on to other senses. We might pick up the complex scents of hay and earth or the drift of warm air on the skin, the shuffle of the wind through stalks perhaps, or the metallic rub of crickets. The field becomes a multi-sensory adventure and we become alive to the richness of the environments around us.

Darkness is shifty, dynamic, of many textures and moods. Think of it like some of the world’s other magnificent habitats, such as desert or grassland, or perhaps the deep ocean. Our experience of them changes with place and weather, with mood and opportunity, with time. Tonight’s dark won’t be the same as last night’s; the dark of a deep cave feels very different from the dark of an unlit room. And as with those other habitats, we still have a lot to learn. Darkness is all around us: theories of dark matter and dark energy suggest that these account for about 95% of the universe. Art, literature, physics, medicine, religion, psychology and philosophy all tackle the dark. But do we really know what kind of thing it is? In biological terms, it exists only as our eyes and minds process the expectation of light. It’s a deficiency, a disappointment – something missing, that’s all.

Is this your experience of the dark? If you walk the fields or lanes at midnight on a moonless night, does the dark around you fall flat and lifeless? Is it just no-light? Or does it seem to have body and essence; does it seem material, tangible even? As darkness closes in for winter, try stepping out to explore, and don’t bring a torch.

Into the Dark by Jacqueline Yallop is out on 9 November (Icon Books). To support the Guardian and the Observer, order your copy at guardianbookshop.com. Delivery charges may apply.