What makes you think they are retires?
The responses that people who are not actively seeking work give in response to survey questions. The BLS conducted a survey on reasons for being out of the labor force and reported the results in their Beyond the Numbers section of its website. On another thread, I gave some more details and noted that just under half (not more than half, as I wrote above) were retired. Of course, some unknown fraction of them could have retired after months or even years of unsuccessful job search.