My best friend is a 3rd generation American born of Swedish grandparents on both sides. She tells me horror stories that her parents told her about the treatment of Swedes in certain parts.
No doubt 99% of the old "it was so difficult" stories arise out of the famines, which happened in the Old World, and conditions on the frontier where just about all the Swedes went to.
On the other hand, there were already people descended from some of the tonier Swedish families who date back to the early 1600s in New York.
Who hasn't thrilled, for example, to find in their ancestry this lady called Maria Von Stockholm!! Makes you a cousin to the Roosevelts, Van Burens, etc., etc., etc.
Not to discredit your GGGrandparents' stories, but they didn't face racial discrimination, nor religious discrimination. Rather, they were young, didn't speak English and were trying to pull a living out of a barren wilderness.
Praise God they succeeded.