You’re right, of course. Like your children, mine had candy only on special occasions but not at other times. They never had cavities, either. But then, this past year, I started buying a type of sticky candy that became their favorite, and the firstborn ended up with one cavity right before heading to college.
Something else I did was have them always rinse their mouths with water within 10 minutes after eating, especially if they had fruit, and especially on the rare candy holidays. As children they did not brush every day.