Justice Minister Andrew Little says inconsistencies in judges' rulings means some people are being held unreasonably behind bars.
"All the judges are doing a fantastic job, they do the job with the law that they've got, they interpret it as they see it," he told Three's The Nation on Saturday.
"But particularly with the District Court that is frankly processing hundreds of cases a week, across dozens and dozens of judges, getting some consistency and some guidelines would be very helpful."
The Justice Minister suggested a guideline for judges' rulings, similar to the Sentencing Council - which was scrapped by the National Government - would help.
"I expect it's more to do with the way [laws are] being applied and enforced as opposed to the law.
"We have to get some consistency in those decisions."
Mr Little says a flawed bail system means some prisoners are serving "way beyond" their parole date, because remedial measures aren't in place.
"People will hit their parole eligibility date, because they haven't done things that the court expects them to do. They can't be released.
"If we're going to put people in prison for having been charged and not yet convicted of a crime, we've got to be reasonable about it.
"We've done the tightening up, but I think what's happened is that we have now remanded in prison a whole bunch of people whose risk of threats to society actually isn't that great."
Mr Little told The Nation he expects the prison population to double in the next 10 years.