Actually that is very true. Ultimately everything we ever experience is the result of our own actions. That's karma. But when we try to isolate one event from another with an expectation that we should be treated fairly in it because it appears that we have no prior contact with the people or things in it we're going to be disappointed since karma can come to fruition from past events that have no apparent connection to the present circumstances. That expectation is neurotic and it is no slur to recognize that it is. It's just reality.
It doesnt really matter (to the original question) if fairness is fundamental or a Human construct, it is still a socially accepted code of behavior, which everyone in a debate, the debators, the questioners, the adjudicators and to some extent the audience, have all contracted into to. Therefore I think its perfectly reasonable for them to have an expectation of being treated according to those set rules.