This is the vital point. In a contest of interpretations, which one wins? Athanasius appeals to an authority other than HIS own or the mere words of Scripture. I will snip a quote of his from
"For not only in outward form did those wicked men dissemble, putting on as the Lord says sheep's clothing, and appearing like unto whited sepulchres; but they took those divine words in their mouth, while they inwardly cherished evil intentions. And the first to put on this appearance was the serpent, the inventor of wickedness from the beginning-the devil,-who, in disguise, conversed with Eve, and forthwith deceived her. But after him and with him are all inventors of unlawful heresies, who indeed refer to the Scriptures, but do not hold such opinions as the saints [meaning orthodox saints or Fathers -- see article at top] have handed down, and receiving them as the traditions of men, err, because they do not rightly know them nor their power. Therefore Paul justly praises the Corinthians, because their opinions were in accordance with his traditions. And the Lord most righteously reproved the Jews, saying, 'Wherefore do ye also transgress the commandments of God on account of your traditions.' For they changed the commandments they received from God after their own understanding, preferring to observe the traditions of men. And about these, a little after, the blessed Paul again gave directions to the Galatians who were in danger thereof, writing to them, 'If any man preach to you aught else than that ye have received, let him be accursed.'" (Festal Letter 2.6)
He asserts that tradition informs and supports his interpretation. The Council ratifies it and from here forward the line is clearly drawn. Orthodoxy is defined. Subsequent councils could then require Bishops to affirm Nicean orthodoxy.
Holding that Athanasius relied solely on Scripture must be understood in the context of what his other writings show HIS understanding to be. Athanasius surely held also with the transmission of apostolic authority, and I expect he would have dealt with your nutty neighbors in a couple of short paragraphs. But, thats for further discussion.
Vainly then do they run about with the pretext that they have demanded Councils for the faiths sake; for divine Scripture is sufficient above all things; but if a Council be needed on the point, there are the proceedings of the Fathers, for the Nicene Bishops did not neglect this matter, but stated the doctrines so exactly, that persons reading their words honestly, cannot but be reminded by them of the religion towards Christ announced in divine Scripture.The Supreme Court of divine truth is that which has been spoken by God, Scripture. That is the only binding authority, and that is the "sufficiency above all things" of which he speaks. But as he says, councils may be needed, and it may help to consult the patristic writings, even though they are not a canon of Scripture, and therefore not anything but persuasive. And in his one example, the Nicene Bishops, does he describe their voice as independently authoritative, or does he relate their derived authority back to its true foundation? The latter, as the force of their words comes only from honest readers being "reminded by them of the religion towards Christ announced in divine Scripture."