"Which Fathers?"
You beat me to it. I'd like to see some specifics on this one. I'm not saying that it isn't true, just that I've never heard this, and that I'd also like to know which Fathers, and a general direction regarding where to read more of what they had to say against icons.
I am no expert on iconoclastic heresy, but it was always my understanding that it was started as a superstitious belief of Emperor Leo III in the 8th century, and not something that was held as a theologoumenna by different Fathers leading up to Leo's prohibition of holy images.
One thing is certain, no Ecumenical Council up to the 8th century (and the one Leo III convened was a Robber Council) showed any tendency towards destruction or prohibition of icons or statues of Christ and the Saints. Certainly, there was no concensus patrum on that issue.