Perhaps the difference is that physics specialists tend to see all other sciences as ‘more soft science’ than physics? Math specialists tend to think physics is “softer’ than pure math, though the more theoretical realms of both tend to be somewhat “detached”, shall we say?
Most scientists seem to consider the soft sciences like psych and sociology as “all soft, no science’.