There is a lower bound. Yes, there are lots of other external considerations that will increase the wasted energy, but you can't go below some minimum limit required to move the mass from point A to point B.
Literally the lower bound is zero.
With friction approaching zero and speed approaching zero, you can get from point A to point B with energy expendeture approaching zero. All the rest is just details.