Bill English says Judith Collins and Jonathan Coleman weren't demoted as "payback" after they challenged him for the leadership of the National Party.
In the days after former Prime Minister John Key announced his resignation, Ms Collins and Mr Coleman put their hands up to replace him - despite Mr Key recommending his Mr English, his deputy, take the reins.
After it became clear Mr English had wide support amongst the party's caucus, both of them withdrew their bids. In Sunday's Cabinet shuffle, Dr Coleman dropped one spot to number seven, while Ms Collins not only slipped two places (to 16th), but lost her Corrections and Police portfolios.
"If you have a look at the two that put their names up against Bill, I think there was a little bit of vindictiveness there," Labour deputy leader Annette King told RadioLIVE on Monday.
"Jonathan Coleman drops down one in the rankings, and Judith drops down two - in fact, she's getting close to the back door."
"It isn't payback time," Mr English later told Summer Breakfast hosts Trudi Nelson and Mike Puru.
"Look, we have been focusing on refreshing the Cabinet - that means moving some people up. They can't all sit in the same seat. So a number of the ministers who've been around a while have found their individual rankings dropped a bit.
"But look, it's what this Cabinet does rather than the intricacies of who's sitting in number nine and who's sitting in number 11."
Ms King praised Ms Collins' performance in the "tough" portfolios she's carried.
"She does polarise people - people either love her or hate her. But they don't have a lot of Cabinet ministers like her, that can take on tough portfolios."
Mr English praised the pair as "competent and hard-working ministers [with] great skills", and didn't think the public would "spend a lot of time" worrying about Ms Collins' fate.
"What they've seen is a pretty smooth transition, and that's a tribute to the culture that John Key left behind."
Mr Key apparently had no input into Mr English's choices, but the second-time Prime Minister says he listened to many of the same people who advised Mr Key, including senior ministers like Gerry Brownlee and Steven Joyce.