I spent part of the day reading some historical accounts of Prince George's county which was very much a southern plantation economy at the time. Quite different from the western part of the state. Conservative also, with the landed gentry believing secession to be a radical (meaning dangerous) idea. Sentiments there were pro-Southern but not to the point of leaving the Union.
Were your ancestors that joined up allowed to return home after serving or were they consripted for the duration?
Had Virginia voted on their bill of secession sooner (Maryland couldn't very well seceede unless Virginia did), Maryland might well have seceeded as well. It would have been a very different war, and possibly have had a different outcome.
Everyone concentrates on cotton, but the big cash crop for Maryland and Virginia was tobacco, which remains a labor intensive crop. No machinery has ever been developed to harvest it successfully. I worked in the tobacco fields as a youngster, and can attest to the tremendous amount of labor involved.
To add insult to injury, the Union established a POW camp no better than Andersonville on a sand spit at Point Lookout, in full view of Westmoreland Co. Virginia. Conditions there were horrible, and many prisoners did not survive captivity.
BTW, just for grins, check out the lyrics of the State Song. It was written during the war by an expat Marylander in Louisiana. The couplet "Avenge the patriotic gore/That flecked the streets of Baltimore" refers to the riots which occurred when invading northern State Militias marched through the city.