The area I grew up in, leafy Wimbledon in south-west London, is bordered by memorials to two towering historical figures. One side dedicates its streets and walls to the legacy of the abolitionist William Wilberforce: the remnants of a house where he lounged with his friends, and the mounting block he used to get on his horse to ride to the Houses of Parliament, still stand.

The other side is devoted to Admiral Horatio Nelson, who having defeated the French navy bought a romantic estate where he stayed with his lover, Emma Hamilton. So many streets, pubs, shops and other local businesses recall this history that local estate agents refer to the area as The Battles.

These two contemporaneous, though contrasting, histories are symbolic of the problems Britain faces in confronting its past. Wilberforce, unquestionably a force for good, helped end, in 1807, Britain’s official involvement in the transatlantic slave trade. But he was not alone. The enormous contribution of black people in Britain at the time – especially activists and writers who were slaves themselves – has no equivalent site of glory, in London or anywhere in the country.

One of the obstacles all these abolitionists had to overcome was the influence of Nelson, who was what you would now call, without hesitation, a white supremacist. While many around him were denouncing slavery, Nelson was vigorously defending it. Britain’s best known naval hero – so idealised that after his death in 1805 he was compared to no less than “the God who made him” – used his seat in the House of Lords and his position of huge influence to perpetuate the tyranny, serial rape and exploitation organised by West Indian planters, some of whom he counted among his closest friends.

It is figures like Nelson who immediately spring to mind when I hear the latest news of confederate statues being pulled down in the US. These memorials – more than 700 of which still stand in states including Virginia, Georgia and Texas – have always been the subject of offence and trauma for many African Americans, who rightly see them as glorifying the slavery and then segregation of their not so distant past. But when these statues begin to fulfil their intended purpose of energising white supremacist groups, the issue periodically attracts more mainstream interest.

Despite student protests, Oxford University’s statue of imperialist Cecil Rhodes has not been taken down

The reaction in Britain has been, as in the rest of the world, almost entirely condemnatory of neo-Nazis in the US and of its president for failing to denouncethem. But when it comes to our own statues, things get a little awkward. The colonial and pro-slavery titans of British history are still memorialised: despite student protests, Oxford University’s statue of imperialist Cecil Rhodes has not been taken down; and Bristol still celebrates its notorious slaver Edward Colston. When I tweeted this weekend that it’s time we in Britain look again at our own landscape, the reaction was hostile.

“I don’t want that nonsense spreading here from America. Past is past, we have moved on,” one person said. Another accused me of being a “#ClosetRacist” for even raising the question. But the most common sentiment was summed up in this tweet: “Its History – we cant & shouldn’t re-write it – we learn from it. Removing statues would make us no different to terrorists at Palmyra.” Therein lies the point. Britain has committed unquantifiable acts of cultural terrorism – tearing down statues and palaces, and erasing the historical memory of other great civilisations during an imperial era whose supposed greatness we are now, so ironically, very precious about preserving intact.

And we knew what we were doing at the time. One detail that has always struck me is how, when the British destroyed the centuries-old Summer Palace in Beijing in 1860 and gave a little dog they’d stolen as a gift to Queen Victoria, she humorously named it “Looty”. This is one of the long list of things we are content to forget while sucking on the opium of “historical integrity” we claim our colonial statues represent.

We have “moved on” from this era no more than the US has from its slavery and segregationist past. The difference is that America is now in the midst of frenzied debate on what to do about it, whereas Britain – in our inertia, arrogance and intellectual laziness – is not.

The statues that remain are not being “put in their historical context”, as is often claimed. Take Nelson’s column. Yes, it does include the figure of a black sailor, cast in bronze in the bas-relief. He was probably one of the thousands of slaves promised freedom if they fought for the British military, only to be later left destitute, begging and homeless, on London’s streets when the war was over.

But nothing about this “context” is accessible to the people who crane their necks in awe of Nelson. The black slaves whose brutalisation made Britain the global power it then was remain invisible, erased and unseen.

The people so energetically defending statues of Britain’s white supremacists remain entirely unconcerned about righting this persistent wrong. They are content to leave the other side of the story where it is now – in Nelson’s case, among the dust and the pigeons, 52 metres below the admiral’s feet. The message seems to be that is the only place where the memory of the black contribution to Britain’s past belongs.

• Afua Hirsch is a writer and broadcaster