Even if Jesus were referring to those who do the works of feeding the hungry and clothing the poor, etc, if all we've done is that which is expected of us, we are unprofitable servants.
Besides, when Jesus comes to set up His kingdom and separates the sheep and goats, we've already returned with Him. We, who are believers who have been forgiven and clothed in the righteousness of Christ, are not the ones being judged. This is not the standing before the Judgment Seat of Christ to be rewarded for our works which is a bonus to already having been saved.
When I am clothed in the righteousness of Christ because of my trusting in Him, then the works Jesus did are credited to my account and God sees those works, so even if I am judged based on works, God is looking not at my own worthless works that I thought were good enough, but at the works of Jesus imputed to me; credited to my account.
If I relied on my deeds to save me, I would be worried, for sure. Who can ever merit salvation? Not me, for sure. Yet i will serve Him to the best of my ability.
Amen! That is what is called being "found in him, not having a righteousness of my own...but that which is through faith in Christthe righteousness that comes from God on the basis of faith." (Phil. 3:9)