I put heavy insulation between the floors over my garage (a built-in garage in my bottom floor underneath my kitchen) since my garage is practically outside air even when the garage doors are closed. My wife and I also went around the house like you said and caulked up around the windows. I had already replaced the outside doors a few years ago anyway and added the rubber felt gaskets around the doors.
How deep are your overhangs?
If you don’t have large overhangs you could add awnings.
Adding awnings to windows on the south side of your house can keep the sun out during the peak hours of the day.
Even adding blinds or some other kind of window treatment can make a big difference in the summer to keep the heat out.
Try doing a search for passive solar construction methods.
One of the other main ideas for passive solar construction is to plant deciduous trees on the south and west side of the house. The idea is they will shade the house in the summer. Then the leaves fall off and the sun warms the house in the winter. The other is to plant evergreen trees on the windward side of the house to block cold winds in the winter.