Look, the characteristics people use to differentiate "races" are usually visible. Although it's probably as great a separation between two groups as is possible, folks with extra red cones and no blue cones are not identified as all that different.
This would have nothing to do with anatomical characteristics whether visual or not. Visual characteristics are just the rule of thumb we use. It works pretty close.