That's completely messed up. If I was called, I'd ask for a name and for whom they work, as I would feel my confidentiality had been breached.
I understand your feeling of breach of confidentiality. Calling and asking who you voted for is so very wrong. It is called a secret ballot for a reason.