There were pro-reb pockets to the south of Knoxville like Monroe County. (Polk voted for secession also, but there's a strong opinion that the vote in Polk was rigged) But you also had counties like Bradley which were generally relative hotbeds for Unionism and violence between the two sides.
I noticed a mention of Sevier County. Sevier was as Unionist of an area as any. The vote against secession in the second election was 1528 to 60.
I appreciate the mention of the books. If y'all are interested in a view from the other side, I recommend Hurlburt's 1866 history of the Civil War in Bradley County. The author was an Indiana man who was located in Bradley in the last months of the war. The guy was obviously a Radical Republican by the way he praises the policies of people like Fremont. His view from the other side as to the legitimacy of the Tennessee secession process is also interesting. And if you had relatives in Bradley back then, he has an extensive division of the sheep and goats, a listing of the Union and Confederate citizens of Bradley County.
I wonder what the first one was? Seriously there were plantations in Sevier County.
Most likely went into Knox, Blount, and Hamilton County. Other counties there were few including in the more northern counties. Although formed before the Civil War not many if any went to Union County, Campbell, Claiborne counties. Cocke County on the mountain maybe had a small community as well but nothing like the more populated areas. Some went into Anderson County but most went into the more developed cities like Knoxville of nearby like the current Alcoa.
There was a community near where I live I'm fairly certain blacks moved into likely immediate Post Civil War but by the early 1960's had moved on. Not due to any harassment but the fact the area was too far to drive for employment and the land was too difficult to work. They had existed there living off the land.
Another indication is in the Knoxville National Cemetery itself the old one not the new one. I worked there at one time and there were quite a few USCT markers.