It doesn't strike me as any different than saying *I don't believe you*.
You seem to be OK with that. You've certainly used it enough yourself.
Why then, do you object then when someone casts doubt on YOUR credibility?
I don't believe you. The difference isn't even subtle.