Yes and as Federalist 45 points out those powers were specific. Every other power, including issues of marriage, abortion, end of life, etc. belonged to the states. Is Madison not clear enough? Especially considering he attended the whole Convention and is considered the father to the document, I believe he knew exactly what the intent was and what it covered.
And, once again, the feds in Article V provide a means to alter powers by amending the Constitution. And did not limit amendments to limiting federal powers. So your argument falls apart both on two fronts - by the plain text of Article V, and by the ratification of the Constitution itself, which was an increase of federal powers at the expense of the states under the Articles of Confederation.