Clothing doesn't somehow become evil because bad people wore it.
Most of the "garb" to which you refer was explicitly commanded by God, in the Torah, to be worn.
Christian clerical attire is only weakly connected to Jewish ceremonial attire anyway. Most of it is directly descended from the ordinary men's attire in the later Roman Empire.
Totally agree!
But the priestly garb was for the Jews - and most things Jewish were superceded by the spiritual reality that came with Christ.