Night after night for some 40 years, the US independent filmmaker, Alan Berliner, has battled with his sleep demons.

He has tried everything to defeat them, including meditation, acupuncture, herbal remedies, "lots of sex" and earplugs.

Recently, he made Wide Awake, a film investigating both insomnia in general, and his affliction in particular. In the film, we watch as night vision cameras capture his nightly torment.

He says the process of making the film "induced, over time, a kind of madness", and his mother suggests on camera that the process is damaging him. "Night after night I am watching myself watch myself not be able to sleep," Berliner says of the 18-month project. "Each night, going to bed was a research opportunity. And that can be tiring."

In the film, he consults a series of medical experts for advice - some of it serious, some of it playful - and in one section, Berliner asks the best way to count sheep.

Despite other films about sleep deprivation, he claims his film is unique, because it is from the "inside out", going "inside my head and using my experience".

He thinks his work is timely, as people are increasingly obsessed with sleeplessness. "Studies show at least one in three people experience insomnia in some way or do not sleep as well as they want to at some stage of their lives," Berliner tells Guardian Unlimited, from his home in New York. He asked to do the interview as late as possible; Alan Berliner does not do mornings.

He delved deeply into the world of sleep. He talks about the "sleep community" and refers several times to his "circadian rhythms" - the 24-hour cycle in the physiological process of a living being. He says sleeping pills are a growing business, worth billions of dollars, and that an annual US conference of sleep experts has gone from involving a few people, up to 8,000.

Berliner says that period novels and journals indicate that people slept around nine and a half hours in the Victorian era. "We are, at least in western culture, averaging seven and a half or so" he says.

"So you see the curve of average sleep is going down and it is never going to go up again there are too many stimulations and pushes and pulls of technology. Once the light bulb hit and opened up the space and the time of the night the idea of everyone getting optimum night's rest went out the window."

A big factor for Berliner, is that we are living in a media saturated age. As the Martin Amis' novel, The Information, puts it, the "information ... comes at night".

The film also reveals Berliner's manic nature, especially during a tour of his work space and his plethora of information. He keeps a vast archive of films - his own and other directors' - and has a set of drawers which play various sound effects when opened. He keeps newspaper cuttings on the subject and internet pages appear throughout the picture.

He says: "I would like to think I am emblematic or symptomatic of broader cultural issues, that I am plugged into the nervous system of western culture and so I have welcomed the enticements and distractions of over information and the bombardment of news.

"The fuel that allows me to make the films in the way I do and use thousands and thousands of images ... that kind of energy and remembering, and logging is also perhaps part of the problem."

He feels he is not alone. "Any of us can stay up at night and Google on the internet. We all have opportunity to take it in ... with so much to do and so many sources, it is a wonder any of us sleep at all ... it is a Sisyphean task."

As Wide Awake progresses, he becomes increasingly sleep deprived, a condition he likens to boiling a frog slowly, - a frog constantly adapts its body temperature so it would not realise it is being cooked alive - "I look dreadful," he keeps saying.

Berliner has a theory that great clangers in history might be down to sleep deprivation. He says: "Former president Bill Clinton is quoted in the film as saying every major mistake he ever made in his life, he made while he was tired. Everyone is engaged in the dialogue of sleep one way or other. It is crucial in whether we function well or how we can cope in our jobs - in times of war, how generals who stay up, night after night, make mistakes."

He talks of the "criminality" of sleep problems, noting lack of sleep cause more accidents than drink and drugs combined. In the film, he observes that in some cultures, the god of sleep is the brother of the god of death.

One indicator of how sleepless we all are is the huge consumption of coffee in the west, he says. "About 85% of Americans drink coffee," Berliner claims, adding that he had not had a coffee for 30 years before he had one in the film as an experiment. "I am buzzing now," he says after. After the filmic coffee experiment, he says he did not sleep, and he won't be going anywhere near the beverage "for the rest of my life". He has a very sensitive bodily system, he adds.

Berliner comes alive and edits his films at night and sometimes does not go to bed till first light. "Like I say in the film, if I have something important to do, I do it at night", he says. He has long thought that his nocturnal working was what gives him an edge and he describes it as "my secret weapon".

He has a love of the night, despite the problems it causes him. "There are other night owls out there who understand the night is a special haven, a magical time, a time of peace that is a kind of secret club."

It is clear his insomnia and the process of making the film created tensions in his family. He partly blames his parents' rowing when he was a child for his sleep problems. He has made personal films before, most notably, Nobody's Business, which was a portrait of his, now late, father, Oscar, and his divorce from Berliner's mother.

He says: "I don't know what the causes of my insomnia are. My grandfather was an insomniac, so that could be relevant ... or it could be just biological. I am not blaming my parents, per se, for it, but indirectly, my parents were fighting or talking and that might be a cause. Maybe the film is a cautionary idea for people: make sure your children are sleeping before you fight."

Berliner's wife, Cherie, gave birth to their son during filming and there are scenes when they clash in the mornings as he tries to adapt to a more "early to bed, early to rise" schedule, so he can see his son and help with his care.

The filmmaker becomes convinced that it is essential for him to oversee his son's sleep and make sure he gets enough. "I am the guardian of his sleep. I don't want him to be an insomniac and Cherie, who was originally somewhat sceptical about my approach, now agrees with me. One of the doctors tells me protecting his sleep is probably the greatest gift I could give him."

Wide Awake will be shown at the Sheffield International Documentary Festival, which runs between October 30 and November 5.