In 2013 we were awarded a Bloomberg Philanthropies innovation prize to create a wellbeing index for the city of Santa Monica in California. It sounded simple. Define, understand and measure what matters most: how people are doing. There was just one thing. No one had ever done it before. There was no model, no textbook and no map. We were heading into uncharted territory.

Guided by a team from different departments (some excited, others sceptical) we set out to tackle this new challenge. We combed through databases and gathered information about crime, transit, libraries, parks, graduation and employment rates. We sorted the data but needed to know more.

So we surveyed our residents, but not by asking them the usual questions about whether they were happy with the city’s services. Instead, we asked about sleep, exercise, and economic worries. Do people smile at you on the street? Will your kids be able to afford to live here? How many fruits and vegetables do you eat? Do you have the time to do things you enjoy? We asked these things because the science and research are clear that the answers matter a great deal. These are behaviours and conditions that can be amplified or muted. They are things that add to or detract from wellbeing.

We then looked to social media. With analysis of likes, sentiments and needs now standard practice among marketers, we wanted to know whether a government could mine this same data for the common good. Twitter provided a treasure trove of information on people’s concerns and when and where they feel better and more connected.

After two years we achieved a critical milestone: in April this year we produced our wellbeing index for the people of Santa Monica.

We learned some things that were predictable and others that were surprising. In national surveys 80% of people report being able to count on neighbours, but in Santa Monicait is just 56%. And while in Santa Monica a relatively high 80% of people vote, many people (29%) feel their civic influence is limited. We pride ourselves on pioneering the farmer’s market movement, but only 25% of residents report eating five fruits and vegetables daily, compared to 55% nationally.

Overall, people are satisfied with their lives and acknowledge they are living in a beautiful place, but a third report feeling stress all or most of the time. One in five young adults reports high levels of loneliness and a third are concerned about missing rent or mortgage payments. With median incomes varying by more than $100,000 across neighbouring zip codes, income inequality is a big issue. More than half of residents do not believe their children will be able to afford to live in the community.

Next steps

So we have an unprecedented amount of wellbeing data on the people of Santa Monica. Now what?

We are still uncovering new layers of information to shape policies, priorities and resources. And we are using the powerful and unique position of city hall to start new conversations with our residents and civic partners. Driving towards smart collective action to improve community wellbeing - well, isn’t that the very purpose of government?

Government isn’t exactly known for moving quickly. That said, 25 years ago sustainability was a new and burgeoning concept and today policies and practices from the sustainability movement influence government at all levels and how we behave as individuals.

As we apply the science of wellbeing to government, we will have a much better understanding about people and communities; we will make better investments in infrastructure and programmes. And our community will demand smart, evidence-based results. We believe this will become the new standard: government of, by, and for the wellbeing of the people.

Sign up here for your free weekly Guardian Public Leaders newsletter with news and analysis sent direct to you every Thursday. Follow us on Twitter via @Guardianpublic