You just can't be too careful, especially when they're young. A major drug bust in a local apartment complex snared 70 parents from our local elementary school. I can't imagine how I'd feel if we'd allowed my son to go to their house to play.
Agree with your contention that controlling access is more difficult as they get older, but with any luck, the foundations we're laying now will pay off in the future.