You are confusing genital herpes with cold sores. I’ve read that 65% of Americans have the herpes virus (the kind that causes cold sores or genital - they can be interchanged), but not everyone has outward symptoms.
Cold sores are HSVI and genital herpes are HSVII. Interesting fact, they don’t always afflict their proper areas.