I believe they are talking about when his brothers arrived needing help would he treat them the way they treated him. I believe did originally pretend not to know who they were and accused them of being spies.
That would be their past catching up to them.