"And I have nothing against the school but it does not match up with many other programs."
It wasn't long ago that teams like Louisville and West Virginia were enduring the same skepticism. Now you list both (as well as Rutgers!!) as readily accepted top-level programs. Even as recent as last year, West Virginia was supposed to get run off the field by SEC champion Georgia, and it was Georgia who was nearly run off the field. Boise State has been doing this for a while, I think they have the most wins in the country over the past 7 seasons, and they matched up pretty well physically against Oklahoma the other day. I can tell you this, that offense would score on anyone in the country. Their level of execution and sophistication put most programs to shame.
Louisville has had good teams for a number of years though none were as good as this one. Rutgers was the pits for decades and improved vastly this last two years. Both have upgraded their schedules and can NOW beat the upgrades. Boise has generally had teams with a lot of victories but has not upgraded their schedule to the extent necessary to gain national recognition. Perhaps it will in the future.
I enjoyed watching them play and almost always root for the underdog but am not convinced they are even close to the top five teams.