Least expensive - depends.
On a variable cost per Kw/h basis, generally, Hydro and the various “renewable” are tops, followed by nuke.
The investment/financing and the various lifecycle plus operating constraints change the picture a lot. Overall Hydro is excellent, if you can get it, Gas is really, really good with modern combined cycle systems - I have been to some new ones and they impress. Of course, gas prices can easily turn against you, and then you can be in a serious pickle, as California was in 2000-2001.
Coal I think is an essential part of a diversified generation infrastucture because it limits exposure to risks inherent in some of the other generation systems. There’s no shortage of coal and prices are stable, and typically lifecycle cost is up there with gas.
“...gas prices can easily turn against you...”
I’ve been involved in some small hydroelectric projects. They are taking the gamble that natural gas prices will go up - as like you said, it costs a lot to get hydro up and running - but then you are assured of a fairly low rate. Of course they also sell the clean aspect of hydro in order to get government grants. Most of the projects don’t go if they can’t get the grants.