It's more than that. It's their future livelihood. The color of the medal could affect their earning potential the next two decades. Hardly "just a game." It's their profession -- not a pickup softball game.
Actually, while I know money and income matter, this is more like the crowning achievement of their lives. If they never hoped to make a cent, the cruelty and pettiness of stealing the symbol which validates the excellence achieved through years of dedication is despicable. Everyone who competes in the Olympics has been working single-mindedly toward that one goal for years, many for most of their lives.
Most will try and fail, losing is the reality with which most will have to deal. Most of those who lose will count themselves winners just to have had the chance to be in the olympics.
But to earn it and have it stolen would be really difficult to deal with.