What your proposing is like saying only the victim and not society can seek justice because only she had a crime committed against her.
You are confusing the issues of personal harm and justice. A person who commits a crime injures a specific victim but also harms society in a general sense. The victim can forgive the perpetrator for the personal harm that he has done, but the victim's forgiveness cannot be applied to the harm that has been done to society in general.
If someone steals my car, it is perfectly acceptable for me to forgive him, even publicly. I may even plead for leniency at his sentencing in court. The court may take my plea for leniency into consideration, but my plea cannot be the sole basis for how justice is meted out in this case.