The Queensland senator insists he was representing local constituents on the interstate trip as his state is experiencing violence from African gangs.

The controversial senator is adamant the rally was attended by "ordinary working people" rather than radicals or skin heads.

"The truth is that attempts to claim that this rally was a 'far right' event appear to be left wing media attempts to distract attention from the purpose of the protest - African gang violence," he said in a statement on Sunday.

"The only people who were doing Nazi salutes were the far left extremists one hundred metres away who came to try to disrupt a peaceful rally," he said.

The senator - who sits as an independent after being booted from the Katter Australia Party following his defection from Pauline Hanson's One Nation - attended the right-wing event on Saturday at St Kilda beach alongside organisers, convicted criminals Blair Cottrell and Neil Erikson.

Several hundred people attended, whith Cottrell and Erikson it was a response to recent incidents in which youths have mugged people along the bay.

Three people were arrested on Saturday at the duelling rallies.

The first was held by anti-racism campaigners ahead of the right-wing event.

Senator Anning uploaded several videos on Facebook with Cottrell, posing for photos and making inflammatory remarks about migration.

Deputy Labor leader Tanya Plibersek said the senator's attendance at the rally was "disgusting".

"I think the vast majority of Australians would be disgusted to think their taxes are paying for an Australian senator to attend an event which seeks to divide, not unite our country," she told reporters in Sydney on Sunday.

While Prime Minister Scott Morrison took to Twitter to blast the "ugly racial protests", he was mum on Senator Anning.

Mr Morrison thanked the hundreds of Victorian police on hand for Saturday's event, while calling Australia the most successful migrant country in the world.

"This has been achieved by showing respect for each other, our laws and values and maintaining sensible immigrations policies," he tweeted.

Labor leader Bill Shorten condemned the event on Twitter but was also silent on Senator Anning.

However Treasurer Josh Frydenberg laid the boot in.

"Fraser Anning's appearance was unacceptable and he should not have participated in this divisive event," the senior Jewish MP told reporters.

He also labelled the use of the Nazi salute as "particularly repugnant and abhorrent".

Mr Frydenberg said it was up to the independent parliamentary entitlements authority as to whether it was an official parliamentary duty expense.

Independent Wentworth MP Kerryn Phelps says the rally should be called out for being a "demonstration by a neo-Nazi group".

Greens Senator Sarah Hanson-Young said the government should refuse to take Senator Anning's vote after his involvement.

Australian Associated Press