War records. Look it up. I'm pretty sure Confederate soldiers stayed in the South, and i'm pretty sure Wounded Union soldiers likewise stayed in the North, and very likely neither group moved around a lot.
Read up on "Soldier's Disease."
The Civil war is what started narcotic addiction in this country. Prior to that, few people knew of, or had access to, large quantities of narcotics.
Most telling of all, the people in charge of the Drug War put forth those figures without your caveats, which would have been beneficial to their argument. What we have is a situation similar to an expert witness for the prosecution giving testimony that favors the defense.
Your star witness hosed your case.