Does he support a filibuster though?
If Chafee does not support cloture, then the retaliation should be swift and terrible: no discretionary money into Rhode Island for any reason, strip him of all his committee assignments, have the RNC and the RNSC endorse his primary opponent and fund the opponent. And simply freeze him out of the caucus.