Prime Minister John Key says he backs the Minister for Primary Industries, despite the MPI's failure to enforce the law when it comes to fish dumping.
That's at odds with the views held by Kiwi fisherman Matt Watson who claims the public's been hoodwinked for years over the issue and the Government must take action.
In 2013, Mr Key said fishing quotas were a much more important issue for Kiwis than spying, while MPI was investigating that behaviour.
MPI did nothing about the illegal dumping of fish then because it was worried about its relationship with the commercial sector and that a prosecution could be acting in "bad faith".
Despite that, John Key insists he has faith in MPI minister Nathan Guy saying: "I think Nathan has done a very good job. I don't think there's any question about the Minister."
Labour leader Andrew Little has a different view, saying: "This is not a ministry you can have confidence in, this is a ministry that needs a rocket under it."
In the past, MPI has publicy blamed problems with its evidence as the reason for not prosecuting. In June, Deputy Director General Scott Gallagher said: "Unfortunately, in the scheme of things, we did not have sufficient evidence to pursue a prosecution."
Also in June, Director of Fisheries Dave Turner said: "We ended up with a legal opinion that we could not prosecute the fishermen for discarding the fish."
Now MPI has accepted that the information was wrong.
Around 600,000 Kiwis fish recreationally every year. Mr Watson, who's currently filming a new TV series in Malaysia, says MPI's failure affects all of those people.
"I think most Kiwis are as upset as I am. We hate to see that waste, those are our fish.
"It's very important to them, and I think all New Zealanders going out and being able to catch fish is just part of who we are and when it's being abused like this it really puts it at risk."
MPI says it's reviewing its management of fisheries.
Mr Watson says that's happened before and he doesn't have confidence MPI will do the job right.