Labour says if the Government wants to be seen as taking the housing problem seriously, their first step should be to sack Nick Smith.
National's suffered a dip in the latest Newshub/Reid Research poll which had them down 1.9 percent, with the Labour/Green alliance picking up most of that figure.
The poll also showed Labour's KiwiBuild policy - to build 100,000 affordable homes over a decade - is popular, with 56 percent supporting the idea and 41 percent opposed to it.
It's is the latest blow for National and is a key difference between the parties in the lead up to the next year's General Election.
Labour leader Andrew Little put the difference in the poll numbers to the perceived lack of action on housing.
He said the first thing the Prime Minister needed to do was to get rid of Dr Smith - the Building and Housing Minister.
"If they do that they will send a signal to New Zealand they are serious about starting again on housing and giving New Zealanders confidence they are a Government that takes the housing issues seriously."
Dr Smith's crime?
"The failure to do anything practical and meaningful, what we've seen from him over the past year or so is gestures which give the appearance of doing something, but he's done nothing practical or meaningful that is accelerating house builds in New Zealand," said Andrew Little.
"Nick Smith has demonstrated time and again he cannot do the job."
While housing is also the realm of Social Housing Minister Paula Bennett and Housing New Zealand Minister Bill English, Mr Little says getting more houses built "sits entirely at the feet" of Dr Smith.
On Monday, John Key said the public knows the answer to the housing shortage is to "build more houses".
"Over the next three years we'll be building broadly 85,000 homes - we've got the biggest building boom in New Zealand's history."
Last week, Newshub revealed National's flagship $1 billion infrastructure fund was cobbled together in just a month.
However, the Finance Minister argued that wasn't the case saying such a fund had been in the pipeline for years.