Well, yes - there's no required number of judicial officers that must exist, either as a minimum or as a maximum. But you really can't get rid of a sitting judge simply by reorganizing in such a way as to eliminate his position. Otherwise, Congress would never need to bother with impeachment of judges - they'd be able to do an end-run around the impeachment process any time they liked, simply by reducing the number of district courts by one and selecting the sacrificial lamb to be downsized out of a job. As I said, though, the courts could be reorganized to eliminate posts that are currently vacant anyway.
There is a big difference between axing a Judge and getting rid of a position. The legislature, by its lonesome, may impeach and convict a judge. The President has a say in the removal of a enacted judicial position. He may veto, by his lonesome, any such end run. Such is the balance of power.