You're right, and I've wondered about that for some time. Can the Secretary of State who is not natural born become President, if the President, Vice President, Speaker of the House, and President Pro Tem of the Senate all resign? He would not be elected but his office would be next in line.
No, because (see my previous post), the eligibility requirements in Article 2 specifically state the requirements to be "eligible to the office," while the 22nd Amendment merely defines how many times one can be "elected to the office."
I believe the Speaker of the House is next in line (and the House can pick whomever they want for that position--it doesn't have to be a Member of Congress)...so in theory Clinton could be chosen Speaker, and the President and Vice President could resign simultaneously, and Clinton could become President. In that case, there wouldn't be any time limit--he could serve the remainder of the term. Fortunately that's never going to happen.
Have you wondered since Henry Kissinger?
Of course the current SecState doesn't have this problem. Secretary Powell was born in New York City on April 5, 1937. His parents were Jamacian immigrants. His immediate predecessor did have that problem though, Madeleine Korbel Albright was born in Prague, Czechoslovakia, on May 15, 1937. She didn't come to the US until she was about 11. Similarly Henry Kissinger was born in Furth, Germany, on May 27, 1923, and fled the Nazies in 1938. Ike's last SecState was also born overseas, Christian Archibald Herter was born in Paris, France on March 28, 1895. (However he would be natural born, because his parents were Americans, there is famous American physician by that name who would have been 30 in 1895, and was likely his father.)