Doesn't make sense unless Ingólfur and his wife Hallgerdur Fródadóttir were the slaves. Or maybe slaves didn't count in Iceland.
Well it does if they had a problem with Hjörleifur in particular, and not with Ingólfur and Hallgerdur.