They forget that if rights come from man, then they can be taken away because in the end there are no rights.
Not forgotten at all. Robert A. Heinlein sums it up quite nicely in Starship Troopers:
"Ah yes, [life, liberty and the pursuit of happiness]... Life? What 'right' to life has a man who is drowning in the Pacific? The ocean will not hearken to his cries. What 'right' to life has a man who must die to save his children? If he chooses to save his own life, does he do so as a matter of 'right'? If two men are starving and cannibalism is the only alternative to death, which man's right is 'unalienable'? And is it 'right'? As to liberty, the heroes who signed the great document pledged themselves to buy liberty with their lives. Liberty is never unalienable; it must be redeemed regularly with the blood of patriots or it always vanishes. Of all the so-called natural human rights that have ever been invented, liberty is least likely to be cheap and is never free of cost. The third 'right'?—the 'pursuit of happiness'? It is indeed unalienable but it is not a right; it is simply a universal condition which tyrants cannot take away nor patriots restore. Cast me into a dungeon, burn me at the stake, crown me king of kings, I can 'pursue happiness' as long as my brain lives—but neither gods nor saints, wise men nor subtle drugs, can ensure that I will catch it."
(Expanding on his statement that "a human being has no natural rights of any nature.")