A Constitutionalist would argue that Congress could legislate according to the powers explicitly enumerated in the Constitution. A libertarian would argue the same.
Actually, libertarians would take issue with the enumerated power of congress to establish post offices which has nothing to do with protecting them from any violation of their "negative rights."
Yes, to that I would add that most libertarians and constitutionalists believe Congress may not delegate its powers to the executive branch, such as what happens in our current regulatory state.