To beleive that you must believe that he feels enslaving Africans is wrong, yet, keeping them and their offspring enslaved once here is acceptable. Don’t think so. He may not have been perfectly specific but he was talking about slavery as a whole.
He said the trade he referenced was in violation of our laws. Slavery in this country wasn’t, for another 50 years.
In fact, Madison himself owned over 100 slaves. Unlike Washington, he never freed any of them, not even in his will.
http://www.montpelier.org/explore/community/enslaved_faqs.php
So it appears that Madison, like Jefferson, talked a good game about the evils of slavery, but didn’t feel strongly enough about it to inconvenience himself financially.
The international slave trade was allowed by the Constitution to be outlawed twenty years after ratification, in 1808. James Madison’s quote is from 1810.