That he may have said, but he also said he had no objections to an amendment that would specifically allow and protect slavery in those territories after they became states.
But the slave holders knew that was unlikely; in any event, their investments were threatened, so they bolted.