Anti-Defamation Commission chairman Dr Dvir Abramovich said a Jewish person saw the man wearing the armband while riding down Atherton Rd in Oakleigh and sent a photo to the commission.
Dr Abramovich called on all tiers of government to unite to ban public displays of Third Reich symbols.
“No one can feel safe when such sickening incidents happen, and we should not stand for this heinous behaviour, especially following Christchurch and San Diego when white supremacist ideology manifested itself in such a deadly way,” he said.
“It is chilling that anyone would so openly exhibit the ugly Nazi swastika — a universal symbol of genocide and evil.
“This open display of hatred, which would have caused enormous distress to a Holocaust survivor, should anger all people,” he said.
“This repulsive display of racism is an attack on the decency of all Australians and violates the memory of the courageous Diggers who fought to defeat Hitler.
“At this time, we repeat our call for federal and state governments to ban the public displays of symbols from the Third Reich.”
Police spokeswoman Leonie Johnson said crimes motivated by prejudice caused serious harm to victims, often leaving them and their communities feeling vulnerable, threatened and isolated.
“Victoria Police is here to help,” she said.
“We do not tolerate violence, including that which is motivated by prejudice, racism or discrimination.
“If you have any information about crime motivated by prejudice you can phone Crime Stoppers confidentially on 1800 333 000.”
While a person cannot be charged for wearing a swastika armband, offences could apply depending on the specific circumstances of each incident.