“So... absence of sunspots means that the overall surface of the Sun is brighter(and hotter).”
Sunspots are an indicator of increased solar activity. Yes, the inside of a sunspot is cooler, but the cause of them is intense magnetic storms. The more sunspots the more bright flares and increased particle output, also average brightness goes up.
During the Maunder Minimum, there were practically no sunspots and here we called it “the little ice age”. The Dalton Minimum also had cooler earth temperatures.
Sunspots peak with increased solar output. We live near a variable star!
Bottom line, we agree that it’s not the sunspot, but the flares and MCE’s that arise around and between them.