Well I'm comfortable riding in sweatpants and a loose T shirt but that might be too casual.
I often color coordinate a ~little~ for organized rides, but I bet not one person will notice what you are wearing :~D
Personally, I wouldn't worry about being too personal. Go the way you are most comfortable.
Around here people show up at these things in every way possible.
Becky