James Madison even said so.
http://www.jmu.edu/madison/emancslaves.htm
To provide a commensurate remedy for the evil, the plan must be extended to the great mass of blacks, and must embrace a fund sufficient to induce the master, as well as the slave, to concur in it. Without the concurrence of the master, the benefit will be very limited as it relates to the Negroes, and essentially defective as it relates to the United States; and the concurrence of masters must, for the most part, be obtained by purchase.
The 14th Amendment was one of the first times the US Government took property without just compensation, and they've been doing it ever since, with less and less qualms every year.
If the slaves were 'taken' by the federal government, the government would 'own' the slaves. This didn't happen - the slaves were freed, not taken. No compensation necessary.