If you want to be polite, don't hang up while they are talking. Hang up while you are talking - then don't answer when they call back.
Don't forget to tell him that he is a drunk.