Hillary Mantel for The Guardian

If you seek his monument, wear a hard hat. For some years Stratford-on-Avon has been a building site while a new theatre grows by the riverside; traffic snarls on the bridge, and puzzled tourists mill glumly outside McDonald’s, wondering where Shakespeare is to be found and why they’re looking for him. There are no letters, James Shapiro says, no diaries, no authenticated portraits except the posthumous. The mystery man is almost 400 years dead, and yet still so powerful that his words can collapse an audience in gales of laughter or make them walk out of the theatre in nauseated shock.

History missed its chances with Shakespeare. His daughter Judith was still alive in 1662, at a time when scholars were beginning to take an interest in his life, but no one collected her testimony. Survivors remembered him: his fellow-actors, his rivals, his sometime collaborators. Ben Jonson laughed at his shaky geography – shipwrecks in Bohemia? He testified to the frantic pace of Will’s invention, and said he loved him “on this side idolatry”. But only a few dubious anecdotes are left. John Aubrey was told that Shakespeare preferred a quiet life; he was no “company keeper”, and if his friends wanted to go on the town he would slide off home, saying he was “in pain”. His grave keeps its secrets, and his monument, Shapiro admits, makes him look more like an accountant than an artist. The absence of frank autobiography is a source of pain to romantics. In his brilliantly readable 1599, a study of a decisive year in the playwright’s life, Shapiro put it like this: “Shakespeare held the keys that opened the hearts and minds of others, even as he kept a lock on what he revealed about himself.”

In that book Shapiro showed that, though we may have no access to the poet’s inner workings, we do know quite a lot about the public career of the man who made a living in London as actor and playwright. We know enough to persuade a reasonable sceptic that there is only one, economical explanation for the plays: Shakespeare wrote them, mostly by himself, sometimes in collaboration. But why do so many people insist that the man from Stratford is an imposter, a fraud, a cover for some more illustrious name? Where did the controversy arise? What are its roots, and how did it grow and sustain itself?

It’s a tale of snobbery and ignorance, of unhistorical assumptions, of myths about the writing life sometimes fuelled by bestselling authors who ought to know better. The trail is strewn, Shapiro says, with “fabricated documents, embellished lives, concealed identity, calls for trial, pseudonymous authorship, contested evidence, bald-faced deception, and a failure to grasp what could not be imagined”. It is failure of imagination that has led successive generations of sceptics to imagine Shakespeare as their contemporary and assimilate his world to theirs, their judgments on his life and times guided by values that are anachronistic. Shakespeare’s supporters, exasperated by the lack of traces he has left behind, have been tempted to forge some; but luckily for later generations, anachronism traps them too. One 18th-century poem, allegedly written by Shakespeare to Queen Elizabeth, described titled ladies drinking tea.

The argument from snobbery is basic to the debate and runs roughly as follows: Shakespeare was a glover’s son from a provincial town, and therefore not very intelligent. He didn’t go to university and had never travelled anywhere, or at least, not that we know. (Gaps in the record are by their nature suspicious, in this worldview.) Since the plays are sophisticated products of a finely tuned and knowledgeable mind, they could only have been written by a courtier with a lofty spirit and superb education, as well as superior experience of life. Step forward Francis Bacon, step forward Edward de Vere, Earl of Oxford. Shapiro concentrates on these claimants, once fashionable; nowadays Marlowe is gaining on them. He extends unfailing courtesy to the Shakespeare sceptics, both living and dead: which is more than the sceptics extend to the man from Stratford. Delia Bacon (no relation) was a 19th-century Baconian who called Shakespeare a “stupid, illiterate, third-rate play actor”. Delia, who died in an asylum, had clinching evidence concerning a Baconian cipher, but refused to share it. Her views – which, as Shapiro says, embrace some provocative and original readings of the texts – were internationally disseminated, and influenced Mark Twain, who thought not only that Shakespeare didn’t write Shakespeare, but that Milton, not Bunyan, wrote The Pilgrim’s Progress. He also thought Queen Elizabeth was a man.

Twain had an admiring and eloquent relationship with the facts of his own life. He didn’t believe that authors could produce work out of what they “only know about by hearsay”, so Shakespeare’s limited life-experience disqualified him. This belief did not stop Twain employing a sort of stunt-writer to prospect for diamonds in South Africa and gather material that Twain could use. The venture was thwarted by the surrogate’s death from blood poisoning, after he stabbed himself in the mouth with a fork. Shapiro keeps an admirably straight face. But it does seem that, once you stop believing in Shakespeare, you’ll believe in anything.

All the world is encoded; nothing is what it appears to be; the authorities are trying to deceive you; there is a gigantic conspiracy stretching from the playwright’s contemporaries to the present-day heirs of the Shakespeare industry, the academics, the actors and the custodians of heritage tea shoppes. In the late 1890s a Shakespeare sceptic called Orville Ward Owen, a Detroit physician, built a decoding machine, a cumbersome apparatus involving rotating drums and a 1,000ft-long canvas sheet, a sort of intellectual mangle designed to wring out key words from texts not only of Shakespeare but of Marlowe, Spenser, Robert Greene and others. “There was,” Shapiro says mildly, “a great deal of interpretive latitude.”

As he conducts us through the pretensions of the Baconians, the Marlovians, the Oxfordians, and on through the latest internet conspiracy theories, larded with pompous quasi-legal language about “reasonable doubt” and “prima facie case”, Shapiro sprinkles his text with glinting, steely facts, about the actors of Shakespeare’s company, about Elizabethan printers and their methods, about what Shakespeare’s manuscripts reveal about how his plays and stagecraft worked. These details, in the chapter which he devotes to Shakespeare himself, are the most riveting part of his book. The contrarian theories, faithfully and respectfully reported, become less interesting as they slide beyond parody. Francis Bacon was the love-child of Elizabeth and Leicester? The Earl of Southampton was the son of Elizabeth and the Earl of Oxford? The Virgin Queen, it seems, was never out of the labour ward. She had a child at 14 by Thomas Seymour; this child was Oxford, who was also her incestuous lover.

Shapiro does not waste words on the preposterous, but he does uncover the mechanism of fantasy and projection that go to make up much of the case against Shakespeare. His book lays bare, too, assumptions about the writing life that come to us from the 18th-century romantics. Those who made Shakespeare a demigod have much to answer for. They played into the hands of those who believed a writer could not also be, as Twain put it, a “grossly commercial wool-stapler”. Shakespeare’s retirement to Stratford causes problems to refined souls. His afterlife, Henry James sniffed, was “supremely vulgar”. But if many of the surviving documents about him concern money, that does not mean that money was all he made.

Shapiro is at his most combative when he engages with the autobiographical approach to Shakespeare studies. Here, William must be saved from his friends as well as his foes. Are the plays encoded episodes from his life? Do the sonnets reveal his soul? Self-revelation, Shapiro persuades us, was not an early modern mode. What Shakespeare demonstrates is the authority of the human imagination. He commands the transpersonal; that is why he is a genius. If the scant facts of his life disappoint, that’s our problem. A genius is also a man who needs to eat. As Thomas Heywood put it: “Mellifluous Shake-speare, whose enchanting Quill / Commanded Mirth or Passion, was but Will.”

Hilary Mantel’s Wolf Hall is published by Fourth Estate.