The idea is to make the Mexicans want to go back on their own, not to just round them all up. And I don't think Mexico can refuse re-entry to its own citizens.
Yes, it can. As a matter of fact, such alternative has already been discussed by Mexican officials in case a massive deportation campaign starts.
There are thousands of deportable aliens in the US who have committed aggravated felonies and will never qualify for a green card.
The US Supreme Court has already decided that such aliens must be released into US and given work permits. If the home country refuses to take an alien, the alien must be paroled as a de facto legal immigrant.