When we learn that household incomes have fallen for the first time in a generation, that radiation levels are rising in the seas off Japan and that regimes in the Middle East are again cracking down, with a vengeance, on their own people, it is easy to see why Samuel Beckett once wrote that the tears of the world are a constant quantity. Many believe this in their heart of hearts – that the world is a cruel place and to believe otherwise is naive.
Yet all of us know of places where people's eyes sparkle and exude hope, and know other places where even the brightest hope is crushed and where people's eyes are dull and despairing. The implication is that although we cannot eliminate grief and sadness, the tears of the world are not a constant quantity – they are, instead, at least in part, a matter of choice.
That is certainly the implication of the mounting interest in happiness that has led the government here, and in countries such as France and Canada, to introduce a series of questions to measure it.
No government has yet taken the next step of reshaping its policies to promote happiness, but the clear patterns that show that levels of happiness tend to be higher in richer, more equal and more democratic countries are strong proof that the tears of the world are not constant.
In two weeks' time, this interest will take shape in a new organisation called Action for Happiness, which will offer access to the mushrooming knowledge about how we can influence our own happiness, or the happiness of our place of work or school, or our community and society. Some of this knowledge amounts to common sense. Exercise helps; other tips are more surprising: for instance, thanking people each evening for the good they have done you during the day serves as a protection against mild depression.
The flood of knowledge from psychology, neuroscience and social research is fascinating and impressive. But there are still huge gaps, and these gaps have radical implications for how we organise innovation and research.
For much of the past century the best route to happiness was thought to be economic growth, growth that was fuelled more than anything by a constant stream of new stuff.
Wealth remains preferable to poverty and the queues for the iPad2 show the continuing appeal of new things, as do the forecasts that by mid-century the roads of India and China will, between them, be making room for 800 million cars.
Yet for all the impact of technology on everything from cutting carbon emissions (through the use of smart grids or hybrid cars) to organising demonstrations (like UK Uncut) it is no longer so obvious that the innovations that matter most are innovations in things. Instead, your happiness is likely to be most affected by a holistic childcare centre for children or a hospice for the very old, a reading group or a bicycle hire scheme, an urban farm or a timebank.
That innovations like this matter in reducing the tears of the world is beginning to dawn on governments, big corporations and foundations alike. These institutions can also recognise that the traditional model that governed how ideas originated in universities or in the minds of great intellectuals does not fit very well with the reality of social innovation. This is much more a matter of trial and error as people try to solve the problems of their own lives.
However, for all its informal roots social innovation can be much better financed, measured and managed. There is a clear parallel with what happened in the last century in the harder" sciences. For the Victorians, a dominant image was of lone eccentrics, working in their shed with tubes of chemicals and suffering the occasional explosion. Brilliant scientists such as Marie Curie worked on their own or in small teams. Science remained a field for gentleman amateurs.
In the 20th century, the very nature of science shifted to the vast laboratories of companies such as General Electric, with men in white coats and clipboards. Du Pont, for example, created a central laboratory called Purity Hall – and by the late 1940s more than half of its sales came from products that had been introduced in the previous two decades. Innovation was industrialised and became the foundation of many of the world's household names, from Toyota to IBM.
Vast sums of public money were also channelled into fostering innovation, nowhere more so than in the US where the cold war prompted the creation of an unprecedented technological innovation machine, stretching from the laboratories of MIT and Stanford, through organisations like the National Science Foundation to venture capital funds and startups. By contrast, innovation in how we live our lives, in social solutions and social ideas, is organised in a much more haphazard way, resembling far more the science of the 1890s than that of the 21st century.
There has never been a shortage of brilliant innovators such as Robert Owen, the founder of the co-operative movement, Florence Nightingale (a great social reformer, as well as a statistician and nurse), Ebenezer Howard (the founder of garden cities) and my own predecessor at the Young Foundation, Michael Young, the founder of dozens of new ventures from Which? to the Open University. But we have not yet made the transition to scale and system that science made in the early 20th century. It is nobody's job to spot the most important emerging innovations, nobody's job to finance them, and nobody's job to help them grow to scale.
Thousands of brilliant projects are experimenting with ways to reduce unnecessary suffering – to alleviate depression or isolation. But it is still a matter of luck whether a great new idea gets funding or support. Governments that invest billions in new hardware still find it hard to accept that they might benefit just as much from systematic innovation in such things as child development or cutting crime.
That this is changing is partly an effect of the rising confidence of the people and organisations working in the field. The award of the Nobel peace prize to Muhammad Yunus was a key moment – his Grameen Bank is a classic example of a social innovation, providing small amounts of credit for poor women in rural Bangladesh.
Most of its elements were not new; what was new was the manner in which it was put together. And, like so many social innovations, Grameen inverted existing power structures: in his case turning peasants into bankers, just as others have turned patients into doctors or students into teachers.
Another factor is the changing shape of the economy. If you ask which sectors will dominate the economy of 10 or 20 years' time the answer is not cars or steel or ships, let alone agriculture. Instead, the industries of "wellness" look most likely to prosper. Health is already a dominant sector in most societies and the one most guaranteed to grow.
Business has been slow to grasp this shift but there are some good examples of business engagement in social innovation. One such is M-Pesa, which uses mobile phones in east Africa to provide an entirely new banking system for poor people, without the costs of a branch network. This is a classic social innovation that meets needs and promotes happiness, but is being run as a commercial operation.
The other new player is government. Barack Obama now has a small office for social innovation in the White House and a $650m education innovation fund, and many countries, from France to Australia, have incubators for social innovation. The European Union is shifting its huge research and development budgets so that they are no longer just about hardware, but also about new services, about citizens' ideas as well as scientists.
Ageing is a good example of the direction of travel. Life expectancy is rising by around three months every year. Science is, rightly, searching for drugs to arrest ageing or to slow the advance of dementia. But the evidence suggests that many of the most powerful factors determining how you age come from what you do, and what you do with others: whether you work, whether you play music, whether you have regular visitors. What is more, many of the most promising innovations try to combine medical support and technologies with this kind of social support.
Social innovation thrives on collaboration; on doing things with others, rather than just to them or for them: hence the great interest in new ways of using the web to "crowdsource" ideas, or the many experiments involving users in designing services. But it thrives more than anything else on hope – a hope born of experience – that the tears of the world are not constant.
Geoff Mulgan is chief executive of The Young Foundation
