I agree with you, but the courts will not. The ability of Congress to grant its permission for things wholesale by statute, instead of doing it on a case by case basis, is firmly “settled law,” as far as the US Federal courts are concerned. To overturn that precedent would disestablish the “administrative state” in its entirety. The Powers That Should Not Be know that. So it will not happen. Too bad, too sad.
There was a 7-2 decision in that case that suggests otherwise. Thomas' concurring opinion was the only one that directly addressed the constitutionality of the Vacancies Act, so it wasn't a critical element of the overall decision in the NLRB case.