When you consider that you had to be free, white, 21, male, and a property owner, not many voters of Southern sentiment were left.
They had either joined up 'across the river' or been arrested.
I spent part of the day reading some historical accounts of Prince George's county which was very much a southern plantation economy at the time. Quite different from the western part of the state. Conservative also, with the landed gentry believing secession to be a radical (meaning dangerous) idea. Sentiments there were pro-Southern but not to the point of leaving the Union.
Were your ancestors that joined up allowed to return home after serving or were they consripted for the duration?