Pretty much everything I have come across says its the mother that must have been born here and the fathers upbringing is pretty much irrelevant, but like I said, Ive always grown up thinking differently. Your thinking was right. The father's citizenship is in no way irrelevant.
Here's a thread from earlier this week that's packed with info:
4 Supreme Court Cases define "natural born citizen"