I'll bet you have some ancestors from Cape Breton Island N.S. or thereabouts. Plenty of black folk up there spoke Gaelic. One of them features in Kipling's novel
Captains Courageous.
My husband is only vaguely Scottish - his mom is half aboriginal Irish and half Bremen German, his dad is a Heinz 57 from central Georgia whose mother was a Dunbar (lowland Scot, not Highland). But he wears his kilt and enjoys himself just the same.
North Carolina got it's nick-name "Tarheels' from the early Scots, mid-1700's.