Solar minimum and maximum have probably existed randomly for 4 billion years. They would not account for periodic ice ages of the past 4M years.
Continental drift may, in the last few tens of millions of years, have interfered with ocean current circulation between the North Pole and equator. With reduced circulation, small changes could have big effects.
Just tossing out a speculation.