Perry was referencing the drastic cuts in military during the Carter years...and the resulting piss poor equipment etc, that ultimately led to the diaster at Desert One (I recommend you read Charlie Beckwiths book on Delta Force, to get the whole story.
My take also. Perry did not claim he was part of the failed rescue effort. Our military was hurt terribly by the Carter Admin. as you say.