John 20 doesn't say that Jesus gave power to forgive and retain sin to the apostles only. Jesus was speaking to His disciples in John 20 and the context indicated both men and women, apostles and those who were not apostles were breathed upon and given authority by Jesus to forgive and retain sins. Additionally, preceding His giving authority to Believers to forgive and retain sin, Jesus said, "Receive the Holy Spirit". It could be argued that "Receiving the Holy Spirit" is a prerequisite for having authority to forgive and retain sin. Assuming this is so, on the day of Pentecost, 120 Believers made up of: men and women, possibly some children received the Holy Spirit with evidence of speaking in tongues.
If you're an apologist for the Roman Catholic church teaching that authority to absolve sins is limited to its priests only, I would respectfully disagree based my read of John 20 and the book of Acts account of the 120 receiving the Holy Spirit in the upper room.
Who else was he speaking to? You are just making things up. Not 'context' look at the actual words