(Why, I know not.)
Well, it seems that at the turn of the 19th/20th century the US Postal service decided to "standardize" town names and ordered all "burg"s to drop the offending "h".
The good citizens of Pittsburgh were pissed. They liked their "h" and wanted to keep it. They therefore petioned the US Congress for an exemption to the stupid order and won their "h" back.
btw, there's a reason Pittsburgh is always at the top or near the top of "Places Rated" and that's because its pretty nice around here and the people are very cool.