Actually, I used to think the same thing. I did a Thesis for my Masters at UT Austin on that very subject. Most Germans were pacifist, and quite a few actually served in the Confederate Army.
I know many pro-Union Germans were involved in skirmished with pro-confederate forces in Missouri.
The "ethnic" element to the Civil War is a fascinating area of exploration. The South had more Hispanics serving than the Union did (mostly Tejano volunteers), with Irish immigrants serving on both sides, despite the fact that many were anti-Lincoln due to the draft.