Here is what he stated in exact context. It is recorded in the Congressional Record:
Every human being born within the jurisdiction of the United States of parents not owing allegiance to any foreign sovereignty is, in the language of your Constitution itself, a natural born citizen.” (Cong. Globe, 39th, 1st Sess., 1291 (1866))
The man authored Section 1 of the 14th Amendment on Citizenship. He clearly knows what the hell he was talking about. It would take a utter fool to not believe that.
Here is the record of his comments from the house floor in Congress. Look at the middle column, second paragraph. Might need to zoom in. I had too.
http://lcweb2.loc.gov/cgi-bin/ampage?collId=llcg&fileName=071/llcg071.db&recNum=332