I'm not sure there's a "right not to listen", but the Constitution does protect Free Speech even if you don't like the message.
You don't recognize the rights of others, only your own.
Harassment and threats are not free speech protected by the constitution.