Over an eight-week period from September to early November, I met every week with a small group of intelligent and thoughtful people to discuss 11 intelligent and thought-provoking books by 11 new authors. The venue was Waterstones Covent Garden and thepurpose was to contribute to drawing up a shortlist for the Guardian first book award. The experience was hugely enjoyable and I feel very lucky to have been a part of it. But it raised more questions than it answered about just what makes a good book, how to evaluate one book against another and whether it is at all possible to come to some sort of fair and objective decision.

As the weeks went by my thoughts centred on four issues. From the beginning we wondered how far we should allow ourselves to be swayed by our own likes and dislikes, or whether to assess each book on more objective criteria such as quality of writing, structure and substance, and to try to consider how far it succeeded on its own terms – even if, as one member of the group put it, the subject matter was "not [our] bag". We did strive for a degree of such objectivity in our discussions, but all of us were swayed by visceral reactions to each book. When it came to the final vote, those whose bent tended to the political opted for books like Felix Martin's Money: The Unauthorised Biography, which furnished ammunition for their interests. Those more concerned with the psychological had psychoanalyst Stephen Grosz's The Examined Life high on their list. The quest for objectivity seemed ultimately to have been a quixotic one.

Sometimes we needed to be reminded that this was a first book award and that one of the criteria was the potential shown for future development by each author. Arguably the risk here is that younger first-time writers, who might be presumed to have more room – and time – for improvement are privileged over the older ones. In practice I think the group simplified this question into the more concrete (if more subjective) one of: "Would I ever want to read anything else by this author?" And this proved to be quite an effective tool of selection.

One of the main fascinations of the Guardian first book award is the challenge posed by comparing fiction with non-fiction and poetry. Over the weeks I began to wonder whether our judging schedule, which entailed reading each book in a week or less, inevitably disadvantaged the non-fiction works. Novels are designed to be read sequentially and continuously, within a relatively short time frame. This does not necessarily apply to works of non-fiction, which might well benefit from being dipped into rather than swallowed whole. Is this one reason why a longlist with equal numbers of fiction and non-fiction became a shortlist of four novels and one non-fiction title?

About halfway through our deliberations it emerged that one of the most important criteria for judging this year's Man Booker prize had been how far the novels on its shortlist offered up new insights on re-reading. I determined to use this as one of my own criteria, since it seemed to me that it could apply equally to fiction, non-fiction and poetry. On reflection, however, it does have some drawbacks. In particular it can prejudice the chances of "genre" fiction. One of the novels on the longlist, Gill Hornby's The Hive, was a rather frothy mixture of romantic comedy, satire and farce. It was an enjoyable one-off read, but I would never want to read it again. So is there, perhaps, a difference between a good book and a good read? And is being a good read a sufficient qualification for winning a literary prize?

Two weeks ago the shortlist was announced and this has forced me to interrogate some of my earlier scepticism about the holy grail of objectivity. Three books on the list – Burial Rites by Hannah Kent, We Need New Names by NoViolet Bulawayo and The Spinning Heart by Donal Ryan – were also on the shortlist of each member of our group and on the amalgamated shortlist of all five of the Waterstones reading groups. Does such a consensus argue that each of them has an intrinsic, objectively identifiable merit? If the winner is one of these three I might become a believer.

The Guardian first book award is announced this evening.

The shortlist

We Need New Names by NoViolet Bulawayo (Chatto & Windus)
Sex and the Citadel by Shereen El Feki (Chatto & Windus)
Burial Rites by Hannah Kent (Picador)
Kiss Me First by Lottie Moggach (Picador)
The Spinning Heart by Donal Ryan (Doubleday)