25 years of raising dairy goats and using the milk in a variety of ways.
Yes having a buck in with the does will taint the milk.
Feeding too much commercial feed and not enough good second cutting alfalfa hay will change the taste.
They need lots of fresh clean water to drink.
And good milking practices, especially cooling the milk down asap.