About half the families in a study of 6000 children struggled to afford healthy food during the first year of life of their babies.
The study is based on the Growing up in New Zealand research tracking a group of 12-year-olds since they were born.
Researchers from the University of Auckland and the University of Otago used information from the longitudinal study in their examination of food hardship in preschool years.
It revealed nearly half of mothers with babies aged nine months were forced to buy cheaper food to pay for other things, with many using food grants or food banks.
The figures shocked lead researcher Dr Sarah Gerritsen from the University of Auckland's School of Population Health.
"These numbers were a surprise, particularly that the 40 percent of Pasifika children and 35 percent of Māori children had used food banks or special needs grants from the government within that first preschool period of the child's life, so up to four-and-a-half years of age."
Links between food hardship and poorer nutrition were also made clear, with children in families using foodbanks 45 percent more likely to have tried unhealthy food and drink in infancy.
Others in food hardship typically had fewer daily servings of fruit and vegetables, and some consumed three or more soft drinks a week by four years old.
"The health implications of that are quite worrying because children at this age are ... setting their food preferences for life," Gerritsen said.
University of Auckland emeritus professor Innes Asher said the findings were alarming and deeply disturbing.
She spent 40 years working as a paediatrician at Starship Hospital and said food hardship could lead to serious health problems.
"In New Zealand we know that we're having a large proportion of children in difficult circumstances being admitted to hospital because of serious underlying infections like pneumonia, serious skin infections."
The impacts of hardship were not always just physical, Asher said.
"We've got families that are stressed, parents not being able to afford healthy food for themselves and having to balance budgets that don't stretch far enough."
"And that stress also gets put on to the children, or they feel it at least, they feel that stress."
Asher was a member of the government's Welfare Expert Advisory Group which concluded two years ago that incomes and benefits urgently need to rise.
She said there had been no substantial change since then.
"This report has said that the food hardship is related to low incomes mainly ... and so it's no good having people on inadequate benefits and inadequate Working for Families [tax credit] having to contact WINZ all the time to try and get a food grant."
Asher said there was also clear institutional racism that needed to be addressed with the report finding Māori and Pacific children far more likely to have experienced all kinds of food hardship.
"This really is very disturbing and it shows we have a situation of institutional racism in New Zealand where we have Māori and Pasifika families disproportionately extremely disadvantaged in this highly stressful situation of inadequate food.
"We really need to get this right."
Paediatrician Renee Liang said hardship in New Zealand had roots in colonisation and was often intergenerational.
She said struggling communities needed to be given the resources to address poverty themselves.
"They already have many of the tools that they need in terms of cultural knowledge, in terms of whānau support, in terms of community support," Dr Liang said.
"And they already know exactly what to do, and this has been shown time again in multiple studies, they don't need to prove anything any more."