You are mistaken. Among numerous others, Earl Warren was a recess appointee.
From Article II, Section 2
"The President shall have Power to fill up all Vacancies that may happen during the Recess of the Senate, by granting Commissions which shall expire at the End of their next Session. "
I stand corrected.