I have a copy of an 1860 letter from an ancient Tennessee ancestor of mine. The letter mentions Brownlow but didn't give an opinion of him. I don't know the politics of this ancestor, but his children and relatives all fought for the Confederates. He was a county official and never had a slave that I know of.
But in the East, he was very popular figure and a high profile personality.