The push back is for loans that have been determined to fail the “reps and warrants” of the seller / servicer contract.
If you sell a loan and represent the borrower had a 720 FICO and it turns out that it was really 660, you have to buy the loan back from the buyer.
So Fannie and Freddie are vilified for not wanting the bad loans. And then, of course, vilified for needing more money from the Feds to cover the bad loans the banks sold them. It’s a tidy system.