Consider all the battles they were in. Seems reasonable to guess that many died. Consider other events (grounds opens up), also, just regualr everyday events- dehydration, getting trampled by a donkey, or whatever. I would guess the rate of birth dropped, and the rate of death increased. Over the period of 40+ years this would greatly reduce numbers.
It's possible that only a handful made it to Canaan, but one would've thought there would be some mention of this.
A million man army would have been very formidable especially before modern warfare. Even if they weren't armed as well, their sheer numbers would have overwhelmed almost any army.