You've made too broad of a statement. SOME contemporaries have believed it. Others did not see evidence of perpetual virginity. You can't say everyone agreed with this interpretation. Best you can say is that some did.
If the notion of custodial marriage were ahistorical, no one would have believed a word of the Protoevangelium.
As to the perpetual virginity itself, indeed, some doubted it then, just like some doubt it now, primarily based on the "brethren" verses.
Actually the best evidence is that the vast majority of contemporaries accepted the Proto-evangelium: it was only in the sixth century, during the reign of Justinian, that the issue of the perpetual virginity of the Theotokos became controversial, and thus a matter of dogmatic definition by the Fifth Ecumenical Council.
The issue before Ecumenical Councils was *never*, 'What ought the Church believe and teach?', but always, 'What has the Church always believed and taught?'