Well now ... I'm no expert here, and you do seem quite certain of what you're saying. Perhaps what I'm about to say is wrong; it's certainly simplistic. But to me, if there's no universe there are no numbers. I think of numbers as nothing but abstractions. They are imposed upon us by the nature of the universe, but that imperative, if gone, would obviate the numbers too. Or so it seems to me. (I'm pinging some folks who know more than I do about this. I'm always willing to learn.)
Numbers aren't exactly my field of expertise either. While one could posit a universe where countability wasn't important for any practical purposes, it seems fairly obvious to me that meaningful observation requires a space that has practical countability. Cognition (in the abstract, not just human) can't develop in a space without all sorts of countable properties.
The person you are replying to qualifies their statement by saying that in mathematics numbers exist independently of any physical existence. Mathematics is a human contrivance, so the statement is correct in that framework. The ultimate question is philosophical -- do abstractions exist independent of physical existence? Or to phrase it another way -- is there really a distinction between the physical and non-physical?