The Prime Minister admits that a new, larger prison at Waikeria might be needed to house New Zealand's record number of prisoners.
"Do I want to build another prison? No," Jacinda Ardern told Newshub Nation on Saturday.
"Do I want extra bed capacity? No. But am I being told that if we had an earthquake tomorrow, we wouldn't have a place to put prisoners? Those are all the things that we're having to grapple with, and that's what Government has to face."
The Government's considering demolishing, rebuilding and possibly expanding the prison, located south of Hamilton, but has also committed to cutting the prison population by a third within 15 years. The cost, alongside upgrades to Mt Eden and Ngawha, has been estimated at close to $1 billion.
Ms Ardern says she's been told the men's prison is "not in a fit state to continue running", and plans to visit the site to see whether it can be repaired.
"What we have here is a question of A - do we need a replacement? And then B, of course, the last Government wanted a huge expansion on the top of that.
"Those are all issues that at the moment we are grappling with."
Last month Minister for Justice Andrew Little told Newshub Nation he was "frankly horrified" by what he saw when visiting the prison and reform was needed.
"You have to ask yourself whether this is a place where someone can go from being bad to being good," he said.
Ms Ardern says there is "a whole range of other remedial action" that could be taken "that isn't building an enormous number of extra beds".
"Do you want to be in a position where you're building extra capacity when, as I say, the crime rate tells us that actually we don't have an increasing problem but we do have an increasing prison population?"
When asked about plans to reduce prisoner numbers - including relaxing bail and parole laws - Ms Ardern said that while the overall prison population is "out of control", the Government does not want to make justice policy decisions "based on bed capacity".
"We're making justice policy decisions on what delivers the best outcomes in terms of safety for the community and reducing reoffending and improving rehabilitation. Those are the basis of the decisions.
"We can't have one drive the other; it needs to be the other way round."