The Senate has the authority to decide.
http://www.scotusblog.com/2016/02/is-a-recess-appointment-to-the-court-an-option/
The Court addressed this in 2014
So fine... what did it decide?