Yams (sweet potatos) are native to south america. There had to be colonization at some point in order for Polynesians to have picked up sweet potatos. It isn’t quite like fruit where you plant it, watch the tree pop up, and remember the fruit. You’d need to get cuttings/roots to plant to propogate it, which would take more than a food raid on the coast to get. Geneticist Bryan Sykes talks about this in one of his books - I think it's the one on the Y Chromosome. Polynesian men travelled to the S. American coast, picked up some crops (like yams) and left Y chromosomes among the Indians there. They're pretty sure it was male-only groups, because they haven't found any Polynesian mitochondrial DNA in S. America (i.e. the kind passed on by women.)
Not just yams (south America to Polynesia) but also chickens. Recently chicken remains were found (pre-Columbian) in South America. Chickens are originally (pre domestication) Southeast Asian Jungle Fowl. So the Polynesians may have not brought women (or if they did they took their babies back with them) but they did bring (and leave behind) chickens.
Check out Thor Heyerdahl's book/documentary movie “Kon Tiki”.