Check out what Christ said to the seven churches in Revelation to see how soon they went off track. For that matter Paul already had a church off track.
Yeah and you know how we know that? Because Paul wrote letters to them.
Now go find me a letter written by a 4th century author telling everyone else to knock off this "saint worship" that was allegedly just creeping into the Church.