I certainly don't agree with you. Would you say it was nobody's business if they saw someone robbing a store or murdering someone, because they were exercising freedom of choice?
Your money is your own, to spend as you like. But not absolutely. If you spend it on an evil cause, others have a perfect right to object. If you used it to hire hitmen to kill someone, for instance, the fact that it's your money would not excuse the purpose for which you are spending it. Regretably, killing babies is legal in this country since a group of imperial judges imposed Roe v. Wade on us. But that doesn't make it right or desirable or a good thing to spend money on.
Robbing and murdering is illegal. Buffett spending his money on whatever is legal is none of my - or of your - business. Feel free to spend your money in the opposite cause, just keep within legal bounds, like he does.