It may well be that some biblical scholars agree but it does not add up to what we are told about what Cain did. And have you noticed how closely the names of Cain's genealogy are to those of the Adam's genealogy?
Notice what is written in Genesis 4:14 "..... Behold, Thou has driven me out this day from the face of the earth; and from Thy face shall I be hid; and I shall be a fugitive and a vagabond in the earth; and it shall come to pass, that everyone that findeth me shall slay me."
Why would Cain be worried about 'everyone' and not his immediate family if only his immediate family were the only peoples on this earth? No other children are recorded by Adam and Eve until after Cain left the area. And I agree we do not know how many other children Adam and Eve had, only the genealogy listed in chapter 5 leading up to Noah.
From a brief article from Answers In Genesis:
“Since the Bible describes all human beings as sinners, and we are all related (“And He has made from one blood every nation of men to dwell on all the face of the earth,” Acts 17:26), the gospel makes sense only on the basis that all humans alive and all that have ever lived (except for the first woman6 ) are descendants of the first man Adam. If this were not so, then the gospel could not be explained or defended.
Thus, there was only one man at the beginning—made from the dust of the earth (Genesis 2:7).
This also means that Cain’s wife was a descendant of Adam. She couldn’t have come from another race of people and must be accounted for from Adam’s descendants.”
for the full article, reasonably succinct - here is a link:
http://www.answersingenesis.org/articles/nab/who-was-cains-wife
They make the very solid theological point that only sons of Adam can be saved. There is only a plan of salvation for those who fell, who were Adam’s posterity.