By your description, the "Muslims" you know are bad Muslims, backsliders, sinners. The "good Muslims" do indeed whip out the prayer rug and pray to Mecca's black stone 5X a day. And it is among that group we find our Islamofascist enemies.
Islam calls for the death of lukewarm Muslims just as it does the infidel, which is part of the reason Al-Qaeda and its symps can live with bombing hotels of Muslims who like to catch waves at the beach and have fun.
Well, I've seen plenty of taxi stands in NYC where the Muslim drivers are "whipping out the prayer rug", somehow I don't find them all that threatening.
But I agree with your premise, and that's why I'm uncomfortable with the efforts to group all Muslims as one and the same.