Surrendering the canal was the eventual result of our support of Egypt during the Suez crisis. Keeping Panama after forcing the British to give up Suez made us look like hypocrites.
I don't remember (m)any Americans saying that in the 20 years between the Suez Crisis and Carter's election. The bigger picture is that empires were falling apart everywhere and we were the original anti-empire power, so we would eventually have to give up our overseas possessions. Suez was part of that anti-empire trend but not the whole story or the chief reason for our giving up the canal.
If Carter hadn't come along we would have held on to the canal longer and negotiated better conditions if we did give it up. Panamanians certainly wanted the canal. There'd been protests and riots in 1964. LBJ dismissed the Panamanians' grievances, and wielded his big stick and Americans backed him up. It was ours, we built it, and we were going to keep it. Later, after Vietnam, Carter came in with a very different point of view.