There are no sour grapes about this inquiry since only the most fragile egos would be bruised by omission from this mysterious category. Does the secret lie in the subject matter? It seems not, although a high proportion of literary novelists are apparently attracted to post-colonial stories, the recent or remote past, or fantasy. Does the distinction lie in literary style? Certainly a few literary novelists develop a style which makes demands on the reader's comprehension, but others write with exemplary clarity and grace. That aside, I doubt whether P.G Wodehouse has ever been regarded as a literary novelist. Today Mr Ishiguro is a literary novelist, and Mr Le Carré is not - or so it appears. I doubt whether either writer is much concerned either way, but why does this difference arise and who decides? Once awarded the accolade, by whatever method, it seems that the distinction is never lost, and however disparaing or hostile the reviews of subsequent books may be, a literary novelist apparently remains so for life.
I should have realised this before reading the thoughts of Thompson's Terror: I remember noting the peculiar way Ian McEwan was described in the list of the Bookseller's top-selling 100 books of the noughties (not available online). He was the highest 'non-genre' novelist (having sold a stunning 27 million books), an adjective that I now perceive was deployed to avoid debate. 'Non-genre'! What a cop out!
And although one senses PD James itches for inclusion, I suspect that in certain circles 'literary' - rather like 'upper class' and 'élite' - has become one of those terms that has edged into pejorative territory. I would also guess that rather like 'intellectual' it's a label that is only ever acceptably used by others, the praiseworthy British reluctance to adopt airs intervenung.
I would stick with my definition - a literary novel is one that sets out to 'reveal the true nature of things' - even if this seems a rather old-fashioned, pre-postmodern pursuit. Isn't 'literary' something of a close cousin to 'scholarly', both involving an informed search for the truth?
This definition is capacious enough to include the better elements of genre fiction: certain thrillers and crime novelists indubitably seek to 'reveal the true nature of things': Le Carré and Ian Rankin spring to mind. However, entertainments which take the world as it is, simply setting out to tell a good story (Dick Francis) or not seriously seeking to reveal the truth (Dan Brown) are excluded. I think the issue of style is dealt with along the way: how can what is true be described without deploying precise and reflective language?
And you can have bad literary novels as long as honest intention is evident. (This may explain why, as James notes, with literary novelists 'the distinction is never lost.')
I would guess I find it relatively straightforward to come up with a definition as I'm not scared of the word élite. And I'm a bit surprised the formidable Dame struggles - but then using literary as some sort of proxy for philosophical seriousness may put her in a bit of an invidious position (I can't comment not having read her books).