As hard as this is to believe, there are more people during those hours looking for cabs than available cabs. This is due to traffic congestion, and the diversity of routing. Many people who have regular destinations make arrangements with car services which are not allowed to pick up at curbside (read that the midlle of 5th Ave.)
You didn't have to qualify yourself as a non-Ny'er, BTW, it was evident that you have never experienced Manhattan grid lock. NY in the morning can be a madhouse.