When I was a child, one of my sisters recounted the following dream: "I was skiing down a nursery slope. All around were gravestones inscribed with 'Little Deer'. At the bottom of the slope I removed my skis and walked up a hill." Both my parents had trained as psychoanalysts - my father was analysed by Sigmund Freud's daughter, Anna - but neither was usually given to interpreting our unconscious motivations. On this occasion, my dad could not resist.
My sister was just coming into puberty at the time of the dream. More Sherlock Holmes than Sigmund Freud, my dad began by asking her how "deer" was spelt on the gravestone. "D-e-a-r," came the reply. His interpretation was that the nursery slopes were the childhood from which she was departing, the graves her persona as a sweet young child, a Little Dear. Walking up the hill was into puberty and adulthood.
I was dazzled. Another universe of meanings beyond normal reality unfolded, and soon after I remember interpreting my own unconscious to myself. Aged 10, as the train carrying me to boarding school for the first time left Paddington, I caught myself repeatedly singing the chorus to Sloop John B by the Beach Boys: "I want to go home, I want to go home, I feel so broke up, I want to go home." I realised that only my unconscious wishes could explain why at this moment I was singing this part of this song.
My interest in psychoanalysis moved beyond such practical indications of the unconscious when studying anthropology at university. I began to read up on the theory and subsequently trained as a child clinical psychologist - all the while insisting on the primacy of the psychoanalytic approach. John and Elizabeth Newson, my tutors, were incredibly forbearing, but they wanted to see the evidence.
At least partly as a result of their tutelage, I developed an interest in empirical research into whether early childhood experience influences adult personality and mental health. And although I subsequently worked at a psychoanalytic hospital (the Cassel in Richmond, Surrey) for eight years and remain broadly sympathetic to the theory, I gradually became more concerned with testing it against relevant scientific studies.
The trouble was, there was very little scientific evidence to prove that interpretations such as the one my father had made were correct. Almost without exception, analysts themselves had little interest in such studies. How could such evidence be amassed? It is only in the past three years, while researching a new book on the effect of the early years, that I have come across the substantial corpus of recent empirical work which, overall, seems to suggest that Freud was more often right than wrong.
This may come as no surprise to some readers. People who have had contact with Freud's ideas, or the therapies that descend from them, often assume that he is widely accepted. But, in my experience, for every one of those there is at least one other who assumes that Freud has long been discredited.
Until recently, the academic psychology establishment's traditional hostility to Freud had been unwavering. It is unlikely you saw the December 2000 special issue of the Psychologist, the profession's house magazine, but it was a remarkable event: an open-minded summary of some of this new scientific evidence about Freudian theory from the organ of official psychology. The issue was guest edited by Bernice Andrews of Royal Holloway College, London, along with Chris Brewin of University College London. Dr Andrews told me candidly, "I'm not actually a great Freud admirer. To a certain extent he was a bit of a charlatan. A lot of what he said about women I find very offensive and I'm certainly not someone who thinks psychoanalytic therapy is something I would want to go anywhere near." But, for all these reservations, Andrews admits Freud did have his virtues. "There's no question he had a brilliant mind and it's a shame so few psychologists are aware of the evidence supporting many of his ideas." For when she and Brewin had finished trawling through the journals, they were pleasantly surprised. Advances in research techniques in the past 10 years had led to important findings that confound more than a century of rabid downgrading of Freud's status.
Freud's ideas engendered vituperation from the start, with the first (somewhat premature) announcement of the death of psychoanalysis made in 1910: Professor Alfred Hoche, a psychiatrist, declared, "One can take comfort from one thing... that psychoanalysis will abate before too long." Fifteen years later, the American psychologist J McKeen Cattell pronounced psychoanalysis unscientific and smeared Freud as "an artist who lives in the fairyland of dreams among the ogres of perverted sex". And so it went on, decade after decade.
From Simone de Beauvoir in the 1940s to Germaine Greer in the 1970s, feminists portrayed Freud's theory as sexist. Psychiatrists, such as Anthony Clare, routinely rubbished it. Academic psychologists, such as Hans Eysenck, poured scorn on the lack of scientific method; so did philosophers, such as Karl Popper. Then, in 1984, one of his own turned on Freud: psychoanalyst Jeffrey Masson declared Freud a coward and a liar for repudiating his own Seduction Theory, which postulated that sexual abuse by parents was the major cause of neurosis. While it is true that Freud's reformulation of his theory - relocating the desire in the child - helped to conceal actual child sexual abuse from exposure for another 90 years, Masson was probably unjust to question Freud's integrity; but since then, there have been books portraying him variously as a cocaine-sniffing, patient-persecuting, intellectual chancer.
Many of the critics are like clever teenagers, picking holes in the logic of the theory or raging against the pathology of its creator (all too often hypocritically using his concepts to rubbish his personality), but never addressing the hard evidence because they are oblivious to it. The last such foray in this paper was by the academic philosopher AC Grayling.
The single most important Freudian idea to have been recently validated is that early childhood experience has an enduring and ever-present influence on our adulthood. Freud's concept of the "transference" of feelings or ideas about people from our childhood on to our everyday adult lives has been confirmed by a series of laboratory experiments, many done in the past seven years by Susan Andersen and colleagues at New York University. Experimental subjects were asked to provide adjectives to describe their parents. Some time later, in a different context, researchers fed back to the subjects a description of a person who shared most of the characteristics they had attributed to one of their parents. For example, a woman who had told of a father who is "interested in politics, athletic and not very happy" would subsequently be told about another man who is "political and athletic". Asked to recall this other man still later, the woman would be liable to say he was political, athletic and not very happy, imposing on the new person all her father's characteristics.
Extending the studies, Andersen has shown that people tend to feel more favourably towards strangers who resemble loved parents or siblings. Subjects were asked to list characteristics of parents. Shortly afterwards, they were told that they were going to meet another subject, of wide experience and expertise, who was described as having the traits they had attributed to their parent. The subjects were likely to feel positively about them. The converse implication is that, for example, if a person resembles a brother or sister beside whom you always felt superior, being in this person's company may make you feel the same way: you project your sibling's traits on to the newcomer, then reinvent yourself as superior by comparison.
Most dramatic of all, Andersen's newest studies show that you are liable to manipulate people to behave in ways that resemble familial originals - that is how powerful is our need to reconstruct the past in the present. Most of us are aware of repeating patterns in our friendships or love life, but the same seems to hold true for a wider range of intimates. So, if you found a parent domineering, you may cause a new person who resembles them to behave like that, too, by baiting them or encouraging them to control you.
Of course, this is a two-way process: other people need you to fulfil their childhood script. Friendship and love, it seems, go beyond two people finding compatibility based on their pasts; it also requires that both feel at home with being fashioned by the other to fit their childhood prototypes.
Freud's ideas about the importance of childhood have also been supported by direct studies of children. Freud put sex at the centre of his explanation of most things, starting from the uncomfortable fact that small children are sexual beings whose sexuality has to be repressed. There is now strong evidence that he was right that three- to six-year-old children are strongly focused on genital sexplay and that this diminishes or is repressed after that age. Be it Doctors and Nurses, Mummies and Daddies or "I'll show you mine if you show me yours", most of us can remember something of the sort in our childhoods.
In 1990, William Friedrich and colleagues from the Mayo Clinic in Minnesota published the results of a study in which they had asked mothers in great detail about the sex lives of their 880 two- to 12-year-old offspring. Two-thirds of the young children were reported to touch their own sex parts, sometimes adopting a seductive manner, often impersonating their parents, older siblings or television characters. Boys favoured public masturbation more than girls, who were partial to exhibitionism.
But these uninhibited days were numbered, very much as Freud predicted. In elementary school-aged children, there was a marked decrease in overt sexuality. Like Adam and Eve, they began to feel uncomfortable if others saw their naked bodies. They demanded bathroom privacy and became self-conscious while changing for swimming at public baths. Hence eight-year-olds were four times less likely than four-year-olds to be reported by their mothers as touching their sex parts or showing them to other children or adults.
Freud explained the self-repression that occurs around the age of five as being the result of "the Oedipus complex". Using as his template the myth of Oedipus, a man who killed his father and had sex with his mother, Freud claimed that this is a universal, genetically-inherited script and that it is the foundation of conscience. He believed that, as small children, all of us long to possess our opposite-sexed parent and to kill the same-sexed one. When small children say, "I'll kill you" or "I'm going to marry mummy", they mean it - fantasy and reality are much less distinct. But, as time goes by, we realise that our same-sexed parent is much stronger than us and, to avoid posing a threat to them, we repress our sexual attraction and identify with their conscience - adopt their moral precepts as our own. In short, conscience is the result of fear and based on that of our same-sexed parent.
While Freud was right about children having a sexuality, even his most ardent empirical fans, the Americans Seymour Fisher and Roger Greenberg, who published a book summarising evidence for Freud's theories, believe that he greatly exaggerated the importance of the Oedipus complex in forming conscience. The evidence suggests that being loved by parents is much more important, combined with consistent rewarding and punishment.
However, it must be admitted that at least some of Freud's oedipal predictions are supported by evidence. The most striking is his claim that we are more attracted by traits of opposite- rather than same-sexed parents when picking sexual partners. Research with couples tends to bear this out. The most notable proof comes from a neglected yet brilliant study, carried out in Hawaii in 1980 by Davor Jedlicka of the University of Georgia.
He chose Hawaii because parents of mixed race are commonplace on that island. He identified a sample of 980 individuals who had all married twice. If Freud's theory that we are more attracted by a repressed desire for our opposite-sexed parent were true, then the Hawaiians should have been more liable to marry partners who corresponded to the ethnicity of their opposite-sexed parent. This turned out to be the case not only for first marriages but for second ones as well. For instance, a woman with a white father and black mother was more likely to marry a white than a black man. Overall, two-thirds of marriages were to the ethnicity of the opposite- rather than same-sexed parent.
Of course, there are alternative, plausible explanations for this finding other than Freud's repressed desire. At its simplest, it could merely be that our opposite-sexed parent is often the model from which heterosexuals learn their ideas of what is attractive. Nonetheless, the finding is consistent with the oedipal predictions, and future research may yet show them to have validity.
An even more fundamental claim of Freud's about childhood is that care received during the first five years is more influential on adult outcomes than subsequent experience. He noted that human beings are almost unique in the animal kingdom in the length of time that they must depend on parents for survival. This is the period when the brain is growing most rapidly. About a quarter of a million connections are forged between brain cells in rats during every second of the first month of their life. Human brains grow with a similarly explosive vigour during the first three years, never again repeated, so that the brain of a two-year-old actually has twice as many synapses (connections between neurones) as that of its mother. Because so much of our mental wiring is developing at such a young age, the effects are lasting and important.
Nevertheless, until recently, the determining role of the first five years was just a hypothesis, one declared a myth by British psychologists Ann and Alan Clarke in an influential book published in 1976. It is only in the past decade that definitive human evidence has arrived to confound them. It is now apparent that the earlier a pattern of negative parental experience occurs in childhood, the greater its likelihood of affecting subsequent psychopathology. In the case of sexual abuse, for example, the earlier the abuse, the greater the number of sub-personalities a survivor is likely to develop in later life and the more profound the damage to the sense of self. Likewise, in a sample of 587 children assessed from kindergarten through into adolescence, those who were physically abused in the first five years of life were significantly more maladjusted in adolescence than those who were first abused after age five. Similar studies prove that the earlier a child is neglected, or had parents who divorced or separated, or suffered financial misfortune, the greater the likelihood of later psychopathology.
The mental illness called personality disorder (people with weak identities, prone to selfishness and unrealistic ideas) has been shown to be heavily affected by childhood experiences. Sufferers are four times more likely than non-sufferers to have suffered childhood maltreatment. The disorder is also many times more common among adults who were in care as children. Forty per cent of the prison population were taken into care at some point in their childhood. This is one of the reasons that a definitive government study of the mental health of 3,000 British prisoners, published in 1998, found that a staggering 80% suffer from personality disorder.
Freud, who considered himself a scientist, predicted - correctly - that it would become possible to measure the electrical and chemical patterns that underpin our mental lives. Modern techniques for analysing brains show that the particular configuration of electrochemistry at any one moment is highly responsive to what is going on around you at that time, but is also conditioned by the experiences you have had during the first few years of life. In the past 20 years it has become clear that we have a pre-existing web of neural connections out of which we make sense of the here and now. This matrix is established by the kind of care we received during childhood: the earlier the experience, the more enduring the pattern. For example, Geraldine Dawson and colleagues at the University of Washington (Seattle) have demonstrated lasting damage to levels of cortisol, the "fight or flight" hormone, in children whose mothers were depressed when they were infants, regardless of whether the mother subsequently recovered from the depression.
Electrochemistry, shaped by past and present psychosocial processes, seems to have a bearing on more obviously physical traits, too: the workings of a girl's body, for example. Studies from places as disparate as Canada and New Zealand find that, on average, daughters whose fathers leave the family home before they reach 10 come into puberty six months earlier than ones from intact homes. If fathers are physically present but emotionally absent, their daughters come into puberty significantly earlier than those with close relationships.
One of Freud's most radical - and famous - claims is that most of our thoughts and feelings are unconscious, rendered so by repression. Although the idea of the unconscious had been around for a long time in the writings of poets and philosophers, Freud was the first thinker to offer a systematic account of what it was, why it existed and how it could explain the symptoms of emotionally disturbed people. Until the 1980s, psychoanalysis was alone among psychological theories in postulating unconscious mental processes, so dominant was the mind-numbingly simplistic behaviourist model developed by the American psychologist BF Skinner. It claimed we should consider only what we could measure - behaviour - and ignore mental life altogether.
Thankfully, today, almost all psychologists accept that many mental processes are unconscious and there are hundreds of experiments to prove it. A very simple example is if subjects are subliminally exposed to the words taxi/cab and hear the word fare/fair. They are more likely to choose "fare" when asked to spell the word, despite having no idea why, explicable only by the presence of an unconscious.
That the unconscious is a repository of repressed experience has also been proved. A (rather hilarious) study in 1996 by American psychologist Henry Adams supported Freud's claim that homophobia can actually be a defence against homosexual desires. Men who believed themselves to be exclusively heterosexual were given a questionnaire measuring their level of homophobia and then shown erotic homosexual videos. All the subjects had their penises attached to a device that can measure small changes in tumescence. Sure enough, the ones previously rated as homophobes were significantly more likely to tumesce when watching the film than the unprejudiced: the more they were conscious of detesting homosexuality, the more they were likely to desire other men.
The mental manoeuvres by which, Freud claimed, we banish intolerable experience to the unconscious has been demonstrated. When American psychologist Paul Cramer made subjects angry - an uncomfortable emotion - by getting them to do frustrating tasks in his laboratory, they made greater use of the defence known as "projection" than anger-free subjects: instead of feeling the anger themselves, they were more likely to attribute it to others.
Freud's notion that "denial" of unpleasant realities is the norm has been royally proven, too. We live in a rose-tinted bubble of positive illusions, highly defended from reality. In a series of experiments over the past 20 years, Shelley Taylor of the University of California has demonstrated that we imagine our friends like us more than they really do and that we tend to assume nasty things, such as illnesses or car crashes, are less likely to happen than is actuarially the case. We also dress up the past to suit the present. When university students are asked to recall their pre-university grades, nearly all of them slightly inflate what they scored, whereas hardly any remember doing worse than was truly the case. We put a positive spin on the future, too, to a remarkable extent. In one study, men who had tested positive for HIV were actually less likely to believe they would go on to develop full-blown Aids than men who had tested negative for HIV.
Another plank in Freud's theory - the significance of dreams - is evidently valid. As Mark Solms, a neuropsychologist at University College London, explains in the Psychologist special issue, dreams are not a random junkyard of chaotically combined thoughts and feelings; they are driven by needs and wishes. A part of the brain has been identified that is responsible for our instinctual appetites, approximating to Freud's concept of "libido". Solms demonstrated that when this bit of brain is damaged, dreaming stops: it follows that dreams do express motivations.
Taken as a whole, this new evidence suggests Freud was a cracking good theorist, imaginative and prescient - the more so, considering his ideas were based almost exclusively on what his patients told him. But the question remains: is psychoanalysis, the clinical practice he invented, effective as a treatment?
At present it is a relative rarity. If you become mentally ill, you are far more likely to be put in the care of a psychiatrist, a medical doctor with an extra training in the administration of drugs or electricity to change "diseased" brains. Many other treatments called therapy have no relation to the Freudian tradition - the cognitive behavioural treatments provided by psychologists, for example.
Few well-constructed studies have been completed that follow up patients several years after psychoanalytic therapy has ended and that accurately measure the - often subtle - effect on emotional health. Nonetheless, there are 26 studies showing that psychoanalytic therapy (once or twice a week) is as effective as other treatments for mental illness. What is more, a large Swedish study has found that full psychoanalysis (four or five sessions a week for at least three years) does have a much greater long-term effect than the less frequent psychoanalytic psychotherapy. British researchers have demonstrated that people with borderline personality disorder (unreal "me, me, me" narcissists, prone to fluctuating moods) did much better than those given conventional treatments if they spent 18 months in a psychoanalytic mental hospital, undergoing thrice-weekly group therapy combined with weekly individual therapy.
Of course, Freud got plenty wrong and many of his ideas are so abstruse that they will never be provable by experiment. My father always maintained that psychoanalytic theory was not scientific because so much of what we experience is untestably subtle and complex. Presented with the new evidence, I suspect he would have said that most of it is only skating on the surface of the inner depths that psychoanalysis plumbs.
That is true. But perhaps we should also be glad that academic psychology is finally achieving some kind of rapprochement with, surely, by far its most important theorist.
· Oliver James is the author of They F*** You Up - How To Survive Family Life (Bloomsbury).
