First of all, they had the Hebrew scriptures which were readily available.
Second, we know that the writings of the apostles were considered to be scriptures and that they were widely circulated amongst the churches:
2Pe 3:15
Think of our Lord's patience as salvation, just as our dear brother Paul also wrote to you according to the wisdom given him.2Pe 3:16
He speaks about this subject in all his letters. Some things in them are hard to understand, which ignorant and unstable people distort to their own destruction, as they do the rest of the Scriptures.Peter elevates Paul's writings to the same level as the rest of the scriptures. Note that people were already distorting the scriptures in Apostolic times, which is why I'm wary of embracing a tradition just because it happens to be old. An old, wrong tradition is no better than a new, wrong tradition.
Third, we know from archeological evidence that there were thousands upon thousands of copies (if only sometimes fragments) of the New Testament writings circulating throughout the churches. They might not have had access to a small, single bound volume as we do, but they certainly had access to the scriptures.
Finally, the early church had the apostles themselves teaching them and the apostles verified that were from God with miracles and signs.