He is speaking of faith and of gifts of the spirit, but the point is still valid, because he is not taking from the ones with plenty, he takes from the ones with little.
Their faith is so little they trusted not in Him. Where as those with much faith, had multiplied that which they had, and God rewards them with even more.
Faith is when we act out on our belief. The third servant did not do even that.