It's feared David Seymour's proposal to change parts of the Human Rights Act would do more harm than good.
The ACT leader wants to decriminalise language that is threatening, abusive or insulting.
"The idea you could be potentially punished for saying something that was offensive or insulting... is something that worries a lot of Kiwis," he told Newshub Nation on Saturday.
Seymour's solution is to repeal a part of the Human Rights Act which protect against "threatening, abusive or insulting" language "likely to excite hostility against or bring into contempt any group of persons... on the ground of the colour, race, or ethnic or national origins of that group of persons".
- Nazis and racists 'offensive' but shouldn't be arrested - David Seymour
- David Seymour faces backlash over Freedom to Speak Bill
He also wants inciting racial disharmony to no longer be a crime, nor for it to be offensive in public.
Matthew Tukaki, head of the Māori Council, says "irrelevant" Seymour's views are dated.
"This inference that it's okay to give licence to people who would think it acceptable to call me the N-word, or 'black something something', it's not acceptable in today's New Zealand," he told Newshub.
The Māori Council is a statutory body established in the early 1960s to represent Māori interests.
"I am sick and tired, as a Māori, of having to put up with people suggesting that this is a debate that we need to have or should have - that we should somehow listen, and somehow find an excuse for why some people call me the N-word," said Tukaki, calling Seymour's Bill a "protection racket for those who think it's their right to call me a n****r".
"I say to David Seymour if you were black and someone called you a n****r, or a fat black bastard or a black c**t, you’d want to have some protection and right of legal challenge."
Who decides what is criminal speech?
Barrister Thomas Harre says it's important to let the courts decide what's criminally offensive.
"By removing the sections that David Seymour proposes, he's taking away the ability for a court to determine whether this is freedom of speech or something that should be punished," Harre told Newshub.
- Why I won't debate those who deny my humanity
- Duncan Garner: Free speech 'vital' in free society
- 'There should be limits' to free speech
Harre says freedom of speech needs to be balanced with the right to be free from discrimination, and Seymour's Bill goes way too far.
"If you're going out of your way to offend somebody, that's not stirring up debate or being constructive in any sense - it's simply offensive."
Harre says the purposes of law is to protect vulnerable groups.
"This is why it's so important that people's rights are balanced - that they have the right to say what we want, but have the right to be protected from hateful or harmful speech."
Tukaki said Seymour needs a reality check.
"This is not the apartheid-era South Africa where he gets to pick and choose what names people call people of colour, like myself."
Seymour's views on racist speech
Seymour's Freedom to Speak Bill will go into the ballot, and will only be debated if drawn out at random. He told Newshub Nation he disagrees with the kinds of speech the current law protects against, but doesn't think they should be criminal.
- Free speech 'not an absolute right'
- Police dismissed Trump fan's abuse at Christchurch mosque as 'free speech', witness claims
- The 41 terror publications you didn't know the Censor banned
Asked by Newshub Nation host Tova O'Brien if he'd be happy with Nazis performing 'seig heil' salutes on the grounds of Parliament or walking down the street using the N-word, Seymour said no.
"That's completely offensive and I think there would be a whole lot of sanctions form that form the wider society, but I don't think the state should be there trying to punish people," he explained.
"I think those people are complete idiots. It is freedom of expression, but it will get exactly what it deserves - which is total contempt and ridicule from all of New Zealand society."
Seymour also wants the Human Rights Commission abolished.