Actually, this is mostly no longer true. Grand Lodges of AF & AM or F & AM in 34 states now recognize their respective Prince Hall Grand Lodges, though that does not necessarily include free visitation. My own Masonic jurisdiction just became the 34th this past winter and visitation requires explicit permission from both Grand Masters.
That said, generally, Prince Hall Lodges are regular and quite legitimate. The usage stems from the Revolutionary War, when white lodges in the US were forcibly divorced from the Grand Lodge in London. The only one recognized at the time was the newly-formed Prince Hall in Boston (named for a man with that name, btw). As the unpleasantness subsided, recognition quickly followed, but mutual recognition did not. That was an extended process which is still going on.
I stand corrected. I just called the Secy of my lodge and he told me pretty much the same thing. He said this came about after I served as Master.(10 years ago). Then they were not recognized by the Grand Lodge.