It’s in Federalist 46, right after your Madison quote: 500,000. The United States population in 1788 was 3 million. You can do the math.