Carter said during the 1980 campaign IIRC that he would "screw the Jews" (for some imagined betrayal, I think).
Late 1980: Cyrus Vance
confirmed to then-New York mayor Ed Koch that Carter, if reelected, would "sell out" the Jews
Jimmy Carter shortly before the 1980 election
2003:
had I been elected to a second term, with the prestige and authority and influence and reputation I had in the region, we could have moved to a final solution
Jimmy Carter, 12-1-2003
I believe you're thinking of James Baker, not Jimmy Carter. Baker once reportedly said in what he thought was a private comment in the White House, "Screw the Jews. They don't vote for us anyway." The highly impolitic comment, made I think in the mid-80s, did a lot of damage to Jewish/Republican Party relations.