It's not a privilege. It is an intelligent judgement made after careful observation.
Other intelligent Catholics have made the same judgement.
So why was Luther not allowed to reject Leo X then?