There isn't "punishment" but a loss of "rewards". What these rewards are, we don't have much to go on, but I think there will be some regret and sorrow over what we could have accomplished for Christ and didn't because of our own wrong motives. There WILL come a time, though, that God will wipe away every tear from our eyes and the former things will be done away with, never to be remembered again. Whatever "crowns" we are given, we cast in the end at the feet of Jesus and proclaim, "Worthy are You, our Lord and our God, to receive glory and honor and power; for You created all things, and because of Your will they existed, and were created." (Rev. 4:10,11)
I shudda read ahead...