But, I will share what I do know, that should help you research it further on your own fairly easily. Pannell/Pannill is prominent enough that numerous, credible and very thorough genealogies are readily available online.
Your relation to James Ewell Brown “JEB” Stuart of Laurel Hill is very straightforward. his mother was Elizabeth Pannill.
Your relation to me, given that we're both descendants to greater or lesser degree (sounds like your direct line whereas it's not mine), would be more of a tangle.
I've got them paternally through Landon Carter, but that is not my direct paternal line it's my paternal grandmother's. And, I've got them maternally through Richard Cox (Coxe/Cocke). It might prove time consuming to determine just what degree of relation there is but given we're both descendants suffice it to say we're distant cousins, regardless of how many times removed.
Is this helpful?
I am finding this extremely interesting, my ancestry. I appreciate any additional information you might have or find.