What was just recently the poorest place in Michigan isn’t Detroit, Flint, or Benton Harbor, but my own Roscommon. Roscommon has its issues, especially with the housing crash and tourism hits, but is also a lot safer than these cities. Not too far behind is Lake County. The only difference is you will get jumped if you dare go to parts of Baldwin, the County Seat. For some reason minorities congregated there even though there wasn't much to steal and destroy to begin with.
Lake County has the highest poverty rate in Michigan by county. I never had a problem in Baldwin. It was a lot like St Helen's except not 99% white. It's not Traverse City or even Gaylord, but it's not Benton Harbor either. I haven't been in Idlewild next to Baldwin but I doubt it's much different. I haven't been there in a few years, but it's a different culture than Detroit.
Idlewild was supposed to be the black vacation spot back when there was de jure segregation. It was becoming a big deal until integration happened and that was it.
One think Baldwin has is the Pere Marquette river going for them.