The search-and-rescue operation is being stood down.
Maritime New Zealand is appealing for sightings of the missing yacht.
A man was seen wading into the water around the same time the clothes were left on the sand.
"This is the worst thing I've ever experienced and I've experienced some pretty horrible things."
Two climbers have been rescued from an altitude of 1800 metres in Aoraki/Mt Cook National Park.
Police say the pair haven't returned to shore.
Police confirmed the news on Wednesday afternoon.