Dubai’s ambition to become the “world’s happiest city” by the end of the decade has suffered a blow with the publication of the latest annual World Happiness Report, which sees the United Arab Emirates slip down the rankings from 20th to 28th place.

The new report, which ranks 156 countries by their happiness levels, also states that “happiness inequality” has increased significantly “in most countries, in almost all global regions, and for the population of the world as a whole.”

In an effort to counter this trend, in 2014 Dubai – one of seven emirates that make up the UAE – launched its own “happiness index”, aimed at collecting data on how government services impacted happiness. Smart devices were distributed around the city – 23 touch-screen terminals positioned in public buildings and linked to government centres – and individuals were encouraged to give feedback by choosing one of three options to register satisfaction or otherwise with their experience.

“Creating happiness is the final result of the smart city agenda,” Ahmed Bin Byat, CEO of the investment group Dubai Holding, told a government summit last year. “Once we are able to manage and meet people’s experiences, we will be able to rise on the happiness index. It is vital because if people are not happy, they don’t stick around in the city; they leave.”

Last month, the UAE’s prime minister and Emir of Dubai, Sheikh Mohammed bin Rashid Al Maktoum, announced via Twitter that his new cabinet included its first “minister of state for happiness”, Ohood Al Roumi. He insisted this was more than a fuzzy feelgood move, and that the initiative would be propelled by “plans, projects, programmes [and] indices”.

One supporter of Dubai’s efforts is Scott Cain, chief business officer at the UK government-funded organisation Future Cities Catapult (FCC), which aims to “accelerate urban ideas to market, to grow the economy and make cities better”. Recently Cain wrote: “Happiness is something Emiratis take very seriously. Following the recent appointment of the UAE’s first minister for happiness and the declaration that Dubai is to be the happiest city in the world by 2019, Future Cities Catapult has been supporting the city in realising its ambition.

“I was recently invited to present the catapult’s view on happiness and wellbeing in the city, and addressed some issues that will be challenging in the UAE environment. It seems they weren’t discouraged as they presented me with an award, which was as unexpected as it was rewarding.”

Cain will be in Dubai later this week for the fourth annual International Day of Happiness on 20 March, which the desert city will celebrate with a series of events. “The highlight will be meeting the Minister for Happiness herself,” Cain says, “and hearing what other cities in the UK and beyond can learn from Dubai’s efforts.”

Some observers have raised eyebrows at the UAE’s “happiness project”, coming as it does amid ongoing human rights concerns. According to Human Rights Watch (HRW): “The United Arab Emirate often uses its affluence to mask the government’s serious human rights problems. The government arbitrarily detains, and in some cases forcibly disappears, individuals who criticised the authorities, and its security forces face allegations of torturing detainees.”

HRW highlights a new anti-discrimination law which “further jeopardises free speech”, and raises concerns about migrant construction workers “facing serious exploitation” and female domestic workers who are “excluded from regulations that apply to workers in other sectors”.

According to Cain: “In many ways Dubai is much more progressive than its near neighbours; many of its senior officials are women including the minister for happiness.” He adds that FCC is “not a public policy advisory group; we follow the lead of UK government”.

While Dubai and the UAE strive for greater happiness, the top of the world happiness league continues to be dominated by northern Europe. Denmark has regained first place, followed closely by Switzerland, Iceland, Norway and Finland. The US is ranked 13th in the new report, two places higher than last year.

The report is produced by the UN’s Sustainable Development Solutions Network. Its co-editor Jeffrey Sachs, director of the Earth Institute at Columbia University, says: “Measuring self-reported happiness and achieving well-being should be on every nation’s agenda as they begin to pursue the Sustainable Development Goals … Rather than taking a narrow approach focused solely on economic growth, we should promote societies that are prosperous, just and environmentally sustainable.”

Follow Guardian Cities on Twitter and Facebook and join the discussion