You are certainly free to be against the World Bank or IMF. But Pat attempts to use our status as the "largest debtor nation" as an argument against them, and that's just stupid.
It is much easier to say what Buchanan did, than to make a real argument. But for him to make a real argument, would be to admit also that, for all their problems, there are also legitimate strategic reasons for supporting one or both organizations.
Buchanan is particularly bad -- worse than useless -- when it comes to seeing how addressing international considerations can (and do) serve the national interest. The cowardly argument he proffers in this instance is at least better than his usual blather ... which isn't much of a compliment.