I've heard your argument on this before, and probably responded similarly before, but the special status you attribute to matter in this respect is arbitrary. ALL scientific concepts (certainly all significant ones) are theory laden in the same or similar respects.