Indeed you are correct. In many respects libertarians are closer to liberals (note the root) than they are to conservatives.
But of course modern “liberals” aren’t really liberals at all. They’re too in love with the central gov’t and its wonder-working powers. Conservatives are the true liberals now. As such, libertarians and conservatives have more in common than modern so-called “liberals” (who thankfully are transitioning to the label “progressive” which more accurately reflects their statist, illiberal inclinations).