Actually, I suggest folks read the entire decision.
The case states:
The "right" to Social Security benefits is in one sense "earned," for the entire scheme rests on the legislative judgment that those who in their productive years were functioning members of the economy may justly call upon that economy, in their later years, for protection from "the rigors of the poor house as well as from the haunting fear that such a lot awaits them when journey's end is near." Helvering v. Davis, supra, at 641. But the practical effectuation of that judgment has of necessity called forth a highly complex and interrelated statutory structure. Integrated treatment of the manifold specific problems presented by the Social Security program demands more than a generalization. That program was designed to function into the indefinite future, and its specific provisions rest on predictions as to expected economic conditions which must inevitably prove less than wholly accurate, and on judgments and preferences as to the proper allocation of the Nation's resources which evolving economic and social conditions will of necessity in some degree modify.
To engraft upon the Social Security system a concept of "accrued property rights" would deprive it of the flexibility and boldness in adjustment to ever-changing conditions which it demands. See Wollenberg, Vested Rights in Social-Security Benefits, 37 Ore. L. Rev. 299, 359. It was doubtless out of an awareness of the need for such flexibility that Congress included in the original Act, and 611*611 has since retained, a clause expressly reserving to it "[t]he right to alter, amend, or repeal any provision" of the Act. § 1104, 49 Stat. 648, 42 U. S. C. § 1304. That provision makes express what is implicit in the institutional needs of the program. See Analysis of the Social Security System, Hearings before a Subcommittee of the Committee on Ways and Means, House of Representatives, 83d Cong., 1st Sess., pp. 920-921. It was pursuant to that provision that § 202 (n) was enacted.
We must conclude that a person covered by the Act has not such a right in benefit payments as would make every defeasance of "accrued" interests violative of the Due Process Clause of the Fifth Amendment.
The decision continues:
This is not to say, however, that Congress may exercise its power to modify the statutory scheme free of all constitutional restraint. The interest of a covered employee under the Act is of sufficient substance to fall within the protection from arbitrary governmental action afforded by the Due Process Clause. In judging the permissibility of the cut-off provisions of § 202 (n) from this standpoint, it is not within our authority to determine whether the Congressional judgment expressed in that section is sound or equitable, or whether it comports well or ill with the purposes of the Act. "Whether wisdom or unwisdom resides in the scheme of benefits set forth in Title II, it is not for us to say. The answer to such inquiries must come from Congress, not the courts. Our concern here, as often, is with power, not with wisdom." Helvering v. Davis, supra, at 644. Particularly when we deal with a withholding of a noncontractual benefit under a social welfare program such as this, we must recognize that the Due Process Clause can be thought to interpose a bar only if the statute manifests a patently arbitrary classification, utterly lacking in rational justification.
Such is not the case here. The fact of a beneficiary's residence abroad—in the case of a deportee, a presumably permanent residence—can be of obvious relevance to the question of eligibility.
This is actually a fairly narrow decision relating to one specific portion of the Social Security Act.
The central issues of Flemming v. Nestor, IMO, are two:
1. Is SS a pension, ie, does it create a property right upon which you can collect at the pre-determined age with the right to seek redress in the courts if Congress denies you the benefits you thought were promised?
The answer is “no.” The court described Social Security as a “social insurance” program:
“The Social Security system may be accurately described as a form of social insurance, enacted pursuant to Congress’ power to “spend money in aid of the `general welfare,’ “ Helvering v. Davis, supra, at 640, whereby persons gainfully employed, and those who employ them, are taxed to permit the payment of benefits to the retired and disabled, and their dependents. Plainly the expectation is that many members of the present productive work force will in turn become beneficiaries rather than supporters of the program. But each worker’s benefits, though flowing from the contributions he made to the 610*610 national economy while actively employed, are not dependent on the degree to which he was called upon to support the system by taxation. It is apparent that the noncontractual interest of an employee covered by the Act cannot be soundly analogized to that of the holder of an annuity, whose right to benefits is bottomed on his contractual premium payments.”
2. The Congress reserved to itself the right to change the Social Security Act (it is in the original legislation, plain as day), and all the court is saying in this instance is that denial of benefits cannot be arbitrary, ie, that they must follow due process in denying them to individuals.
Going through the legislative process and setting the standards by which people can make claims upon the system would meet that test.