There is no evidence that this is exclusive to a specific set of Christians. It wasn't just a few who received the gifts of the Holy Spirit as evidenced in Acts 2:4, Acts 10:47, and 1 Cor. 16:9 and especially in 1 Cor. 14:26. Passing on of the Holy Spirit is through laying on the hands as in Acts 8:17 and Acts 19:6. So if laying on the hands and receiving the Holy Spirit is evidence of authority, then all Christians have this authority.
For example, the incidents in Acts 8:17 and Acts 19:6 are examples of the Sacrament of Confirmation, a Sacrament still celebrated today as it was back then, to whit: By a Priest, Bishop (or higher) laying his hands on a "disciple" (or believer), to seal them with the Gift of the Holy Spirit.
In Acts 13 however, we read how the Holy Spirit asked that "Barnabas and Saul be separated for the work I have for them". This "separation" can't be Confirmation, as that was already accomplished when Ananias layed his hands on Saul in Acts 9. Indeed, this "separation" was to do specific work the Holy Spirit had for them, work that obviously needed another Sacrament, the Sacrament of Ordination, which also involves the laying on of hands. This "separation" is accomplished in verse 3 where hands were lain upon them. Note, it wasn't until after this that Saul's name changed to Paul in Scripture, thus signifying the important event that had occurred in verse 3.
Further reading: Apostolic Succession (specifically point II) and Confirmation