He said place was the most important criteria of allegiance, and what applies in the United States. Self evident.
And yet his very next words are:
"Mr. Smith founds his claim upon his birthright; his ancestors were among the first settlers of that colony."
An appeal to blood. Why argue blood if place is the only thing of importance?
I think the merit of the question is now to be decided, whether the gentleman is eligible to a seat in this house or not, but it will depend on the decision of a previous question, whether he has been seven years a citizen of the United-States or not.From an attention to the facts which have been adduced, and from a consideration of the principles established by the revolution, the conclusion I have drawn is, that Mr. Smith, was on the declaration of independence a citizen of the United-States, and unless it appears that he has forfeited his right, by some neglect or overt act, he had continued a citizen until the day of his election to a seat in this house. I take it to be a clear point, that we are to be guided in our decision, by the laws and constitution of South-Carolina, so far as they can guide us, and where the laws do not expressly guide us, we must be guided by principles of a general nature so far as they are applicable to the present case.
...
It is an established maxim that birth is a criterion of allegiance. Birth however derives its force sometimes from place and sometimes from parentage, but in general place is the most certain criterion; it is what applies in the United States; it will therefore be unnecessary to investigate any other. Mr. Smith founds his claim upon his birthright; his ancestors were among the first settlers of that colony.
It is well known to many gentlemen on this floor, as well as to the public, that the petitioner is a man of talents, one who would not lightly hazard his reputation in support of visionary principles: yet I cannot but think he has erred in one of the principles upon which he grounds his charge. He supposes, when this country separated from Great Britain, the tie of allegiance subsisted between the inhabitants of America and the king of that nation, unless by some adventitious circumstance the allegiance was transferred to one of the United States. I think there is a distinction which will invalidate his doctrine in this particular, a distinction between that primary allegiance which we owe to that particular society of which we are members, and the secondary allegiance we owe to the sovereign established by that society. This distinction will be illustrated by the doctrine established by the laws of Great Britain, which were the laws of this country before the revolution. The sovereign cannot make a citizen by any act of his own; he can confer denizenship, but this does not make a man either a citizen or subject. In order to make a citizen or subject, it is established, that allegiance shall first be due to the whole nation; it is necessary that a national act should pass to admit an individual member. In order to become a member of the British empire, where birth has now endowed the person with that privilege, he must be naturalized by an act of parliament.
What was the situation of the people of America when the dissolution of their allegiance took place by the declaration of independence? I conceive that every person who owed this primary allegiance to the particular community in which he was born retained his right of birth, as the member of a new community; that he was consequently absolved from the secondary allegiance he had owed to the British sovereign: If he was not a minor, he became bound by his own act as a member of the society who separated with him from a submission to a foreign country. If he was a minor, his consent was involved in the decision of that society to which he belonged by the ties of nature. What was the allegiance as a citizen of South-Carolina, he owed to the King of Great Britain? He owed his allegiance to him as a King of that society to which, as a society he owed his primary allegiance. When that society separated from Great Britain, he was bound by that act and his allegiance transferred to that society, or the sovereign which that society should set up, because it was through his membership of the society of South-Carolina, that he owed allegiance to Great Britain.
Mr. Smith being then, at the declaration of independence, a minor, but being a member of that particular society, he became, in my opinion, bound by the decision of the society with respect to the question of independence and change of government;
Madison concludes:
So far as we can judge by the laws of Carolina, and the practice and decision of that state, the principles I have adduced are supported; and I must own that I feel myself at liberty to decide, that Mr. Smith was a citizen at the declaration of independence, a citizen at the time of his election, and consequently entitled to a seat in this legislature.
As you can see, the topic at hand had nothing, not a single thing, to do with "natural born Citizen's."
In fact, the person's eligibility in question (Mr. Smith) was a "a citizen of the United States, at the time of the adoption of this Constitution" which is in contrast (via the OR clause) to "natural born Citizens."
The context of Madison's words surround the question of eligibility to the House of Representatives of Willaim Smith, who was challenged by Dr. Ramsay.
House of Representatives, Contested Election
22 May 1789Annals 1:397--408
Resolved, That it appears to this House, upon full and mature consideration, that the said William Smith had been seven years a citizen of the United States, at the time of his election.
Mr. Smith.--As the House are inclined to hear the observations I have to make, I shall begin with admitting the facts stated in the memorial of Doctor Ramsay, hoping the House will excuse the egotism into which I am unavoidably drawn. I was born in Charleston, South Carolina, of a family whose ancestors were among the first settlers of that colony, and was sent to England for my education when I was but twelve years of age. In 1774, I was sent to Geneva, to pursue my studies, where I resided until 1778. In November, that year, I went to Paris, where I resided upwards of two months in the character of an American gentleman. Immediately on my arrival there, I waited on Doctor Franklin, Mr. Adams, and Mr. A. Lee, the commissioners from Congress to the court of France, as a citizen of America, and was received as such by them. In January, 1779, I left Paris for London, whither I went to procure the means of embarking for America, from the gentleman who had been appointed my guardian by my father when I was first sent to Europe in 1770, and from whom alone I had any hope of obtaining such means. But in this endeavor, I was disappointed, and remained some time in England, with the hope of receiving remittances from Charleston. Here again my expectation was defeated. The rapid depreciation of the continental money rendered the negotiation of money transactions extremely difficult, and thus I remained till the fall of Charleston. I took this opportunity of studying the law, but could not be called to the bar, because I had not taken the oath of allegiance to Great Britain, which is a necessary qualification. After the surrender of Charleston, the whole State of South Carolina, fell into the hands of the enemy, and it was impossible at that time to return. No sooner, however, did I acquire the means, and an opportunity offered, than I prepared myself to go back to America. I quitted London for that purpose in October or November, 1782, not in a vessel bound to Charleston, then a British garrison, and which I certainly should have done, had I considered myself a British subject, and which would have been most convenient, as there were vessels constantly going from London to Charleston; but I travelled to Ostend, and there embarked in a neutral vessel bound to St. Kitt's, from whence it was my intention to proceed to a Danish island, and thence to some American port in North Carolina or Georgia, from whence I could reach the American camp. In the beginning of January, 1783, I sailed from Ostend, but was detained a considerable time by contrary winds, and in the middle of the month of February, was shipwrecked on the coast of England, and was obliged to return to London in order to procure another passage. These circumstances unavoidably prevented my return to Charleston, until some time in November, 1783.
...
http://press-pubs.uchicago.edu/founders/documents/a1_5s13.html
There were questions of Mr. Smith's allegiance, because he had been in Great Britain and parts of Europe during the revolution...and weather or not he meet the 7 years residence requirement for the House of Reps seat.
That quote of Madison's had nothing to do with who they considered "natural born Citizens."