This shouldn't be complicated, if you put a single unit patch on something personal then people are naturally going to think that you served in that unit, if it is part of a collage, or collection of various patches, then they would see it as such.
If you are the type that sews patches onto your gear, you should have an eye for that kind of thing already.
I apologize if I wasn’t clear. The patches don’t distinguish unit information, citations, or anything other than, for instance “Scout” and “Tunnel Rat,” as my uncles were a scout and tunnel rat, respectively.