Pro-Brexit Yellow Jacket protesters block roads in Westminster over the weekend

Westminster protests: Free speech requires that we shut down the far-right

If you visit the green outside parliament this afternoon, there's a strong chance you'll be harassed and abused. That's especially true if you're a Remainer, or if your skin is not white, or if you're a woman. A small group of far-right thugs, wearing the yellow jackets popularised in the Paris demos, are bullying and trying to intimidate the people going to talk to the broadcasters.

This has been the case for a few weeks now. What started as a small, peaceful Remain protest outside parliament grew to include mainstream Brexiters, who were equally peaceful, and then, crucially, a few aggressive pro-Brexit men in yellow jackets. They've been chanting abuse, particularly at Sky, and particularly at presenter Kay Burley, political editor Faisal Islam, Corbyn-supporting commentator Owen Jones, Remain campaigner Femi Oluwole and Tory Remain MP Anna Soubry. What a coincidence. Two women, a gay man, a black man and an Asian man. I wonder what it is about these people in particular that a bunch of fascist thugs hate so much?

Fifty MPs have now written to Metropolitan Police commissioner Cressida Dick expressing "serious concerns about the deteriorating public order and security situation in and around the exterior of the parliamentary estate". They're right to do so. No-one has been hurt yet, but the videos make it quite clear that there is a sense of barely-restrained violence among the men there. It must be tackled now, before someone is assaulted.

These are delicate free speech issues. You have to find the balance. Everyone has a right to go down to parliament and state their case. Before the yellow jackets arrived, the atmosphere there was strangely British and reassuring – Remainers and Brexiters standing side-by-side, chanting and cheering when a car went past hooting, despite it being impossible to make out which side it was in support of.

It's a really grave thing whenever we stop someone's right to protest. But there is a limit to protest, a liberal limit: you cannot protest in a way which limits the rights of others to protest. The behaviour we're seeing here is so aggressive and threatening that most people would choose not to visit the area if that's what they're going to face. That goes especially for women and minorities, who are likely to be specifically targeted.

The police response needs to be balanced too. Too often the police do too little and then far too much. The videos from yesterday showed a large but completely ineffective police presence, as they basically stood back and allowed the harassment to take place. This encouraged the behaviour, as it was clear there would be no repercussions.

There are plenty of bits of legislation they can use to clamp down on this stuff. The most obvious is Section 5 of the Public Order Act, which bans "threatening [or abusive] words or behaviour". This is a terrible bit of law – it is really broad and has been systematically misused by the police as a catch-all law for arresting peaceful demonstrators. But this does seem a moment in which it might actually be applicable.

It's understandable why police have been cautious. Things are tense now, politically. And they are set to get worse, probably much worse, over the next 80 days. It's important not to allow the situation to flare up. We also don't want the area outside parliament to become a high-security zone.

But the test authorising intervention – which is that normal people would feel intimidated from exercising their right to protest – has clearly been reached. Something should be done. It should be small, targeted and light-touch. It should not involve flooding the area with police.

The most important response, however, is not a police one, but a cultural one. It's about how we all talk to one another.

Neither side of the Brexit debate should dare to exploit what's happening down there. It would be easy for Remainers to say that this group reflects the far-right heart of the Brexit project, but that is clearly false. The Brexit-supporting protests in the area have been mostly very polite and peaceful.

Some leading Brexit figures have tried to either justify the aggression or use it for their own gains. Tim Montgomerie, the creator of ConservativeHome and a man who used to be a respectable political commentator on the centre-right, sent a disgraceful tweet this morning suggesting Soubry deserved the harassment she received.

Steve Barclay, the Brexit secretary, suggested on Radio 4 today that the aggression showed why there should not be a People's Vote. This is an incredibly dangerous and irresponsible argument to make. It essentially states that we should do what aggressive far-right demonstrators tell us to do. Coming from an actual member of the government, this encourages them to abuse, insult, harass and threaten more MPs and journalists.

Both these comments form part of a wider Brexit narrative, cynically cultivated by Nigel Farage and others, in which the prospect of street violence is encouraged if they don't get their way. They pretend to be warning of street violence, but in fact, with a not-so-subtle wink, they are encouraging it. Both Barclay and Montgomerie should be ashamed of their contributions and retract them immediately.

But the truth is we all need to think about the way we behave and what kind of culture we're creating. These are divided times – sure. Everyone knows that. It's not going to go away. But there are ways of placing limits on the social norms that frame that division. The first norm we should maintain is an aversion to the concept of 'traitor' – the idea that our opponents are not patriotic, or secretly hate or are trying to undermine their country, or are in league with foreign agents against the interests of the country.

This has always been a favoured piece of rhetoric for authoritarians and totalitarians. It was common in the 1930s. It is a core part of the anti-semitic armoury. It has made its way into our political culture slowly and surely over the last few years.

It is very common online. Remainers are often told they are loyal to Brussels over Westminster, for instance. But it is also common from mainstream MPs, or on the comment pages of respectable newspapers and the front pages of tabloids. The very phrase 'Enemies of the People', emblazoned over the front of the Daily Mail after the Article 50 judgement, is an example of this. It must stop. It should be called out wherever it takes place, however subtly it is done.

British politics is a series of disagreements over what is best for the country, conducted by people who all want the best for their country but are divided over how to achieve it. Everyone should be able to sign up to that proposition. Anyone who does not is poisoning the way our culture operates and helping to create the ugliness we're seeing out there on the street.

Ian Dunt is editor of Politics.co.uk and the author of Brexit: What The Hell Happens Now?

The opinions in politics.co.uk's Comment and Analysis section are those of the author and are no reflection of the views of the website or its owners.