Even if Congress is supposed to deal with tariffs, the Supreme Court has to allow those the are negotiated to continue. We cannot afford to return the tariff money back to every country that paid it. That would be a disaster.
Ultimately the Trade Act of 1974 gives POTUS broad authority to set tariffs. At some point SCOTUS will uphold that authority.