And yet his very next words are:
"Mr. Smith founds his claim upon his birthright; his ancestors were among the first settlers of that colony."
An appeal to blood. Why argue blood if place is the only thing of importance?
Because if you read what you have posted very carefully, you'll notice that Madison mentions that Mr. Smith founds his claim on birth. Not that Madison necessarily agrees with an appeal to blood.