Yeah they did. Silver has broad authority under the NBA's constitution and bylaws to suspend and fine an owner for conduct detrimental to the NBA. Sterling admitted it was his voice on the recording in which he made racist remarks. Even if the recording was unlawfully created under California law Silver is authorized to punish Sterling based on the recording's impact on the league. The argument is that Sterling's comments have harmed the NBA and its relationship with players, sponsors and fans. Given all that happened, Sterling doesn't have a viable argument that his conduct didn't do that.
There is nothing ephemeral about this. The legalese is clear. I know that your feelings are hurt and you resent rich guys and you want to see him forfeit his property in reparations for your hurt feelings, but it's not about you. It's about the rule of law, and the rule of law definitely applies here.
What conduct? Use the “but for” paradigm. Had the tapes not been released, there would have been no harm to the league, and he had nothing to do with the release of the tapes.