A couple of Sundays ago, probably while I should have been reading Ed Smith's new book, Luck, I flicked on the television to watch the opening exchanges – "just the first 10 minutes" – of the Premier League match between Spurs and Man United. At half-time, I hadn't moved and Tottenham, despite totally dominating the action, were one-nil down to the champions, through a Wayne Rooney header just before the break. As a neutral, I can say it was undeserved, even unlucky. They wuz robbed.

Harry Redknapp, the Spurs boss, agreed. "We played so well first half, and to come in 1-0 down, it was unbelievable," he huffed, after the eventual 3-1 defeat. "Sometimes you get the breaks when things are going well, and they got the breaks."

Talking about your luck – either good or bad – is not really the done thing in sport. As the toothy Notts County manager Jimmy Sirrel once noted: "The best team always wins and the rest is just gossip." In this Olympic year, it is particularly so. The success of our cycling team in Beijing four years ago was achieved through eliminating chance, as much as humanly possible, and it is that ethos which now pervades all of Team GB as they strive for a golden summer. Work hard, reduce mistakes, control your destiny.

This environment makes the central contention of Luck: What It Means and Why It Matters rather unfashionable, and Smith knows it. Early on, he cites Malcolm Gladwell's now-ubiquitous theory from Outliers that it takes 10,000 hours of practice to be successful, even in Mozart's case; he also references Bounce: The Myth of Talent and the Power of Practice by Matthew Syed, which states that geniuses are "made" not "born" – even Roger Federer. Smith admits that in his past life – he was an England cricketer before becoming a journalist for the Times – he had little time for the concept. "I thought talking about luck sounded like an admission of weakness," he writes.

So, we are in an era where "luck is in retreat… gradually supplanted by a more scientific understanding of the world". An era where books about self-help – "anti-luck", he calls it – make $10bn a year in the US. Smith's mission is to convince us that factors outside our control are just as important as those that we directly influence.

Does he succeed? He certainly sets out a compelling case. In his previous book, the sporadically excellent What Sport Tells Us About Life, Smith chose a massive topic – two massive topics, really: sport, life – and spread himself a little thin. This time, he has a tighter brief, and he writes with clipped authority on his home turf of cricket, on politics and on the financial crisis. He remains fond of a tangent – the Azande tribe from South Sudan have a cameo, for example – but he never loses sight of his central argument.

And luck is a wonderful subject. It allows Smith to spend time with a 94-year-old who flew Hurricanes in the Battle of Britain and who recounts numerous brushes with death, including a parachute failing to open, with effervescent charm. He also talks DNA with James Watson, one of the molecular biologists who discovered it, and randomness with author and trader Nassim Taleb; he even holes in one with golfer Colin Montgomerie. It enables him to theorise why football is the world's most popular sport – because, to his mind, luck plays an integral but not overwhelming part in so many matches.

Going deeper, Luck proposes that we have now become a fortunocracy, rather than the meritocracy that politicians so often jaw-jaw about. This has interesting implications: when we are told that anyone who works hard enough can be a genius, what happens if we do not take the world by storm? Smith is a powerful advocate for genetic, innate skills and argues that we massively underestimate the impact of chance events. "The intervention of luck is like a boulder that diverts the course of a stream," he writes, "the course is changed – and stays changed for ever, whatever happens downstream."

Smith's best case study is himself, and Luck comes alive when he is giving an unvarnished analysis of his own career. As a cricketer, he had many strokes of good fortune (a private-school education; injuries to rivals that led to his England selection) and bad (an incorrect LBW decision in his third and final Test match that, under today's rules, he could have challenged), but he didn't think like that at the time. It took a career-ending ankle injury in 2008, which at first seemed innocuous, for him to even consider the subject and write this book. That turned out to be his good luck, and ours too.