I believe you're thinking of James Baker, not Jimmy Carter. Baker once reportedly said in what he thought was a private comment in the White House, "Screw the Jews. They don't vote for us anyway." The highly impolitic comment, made I think in the mid-80s, did a lot of damage to Jewish/Republican Party relations.
See 48. Apparently a topic Baker and Carter saw eye to eye on.