In the Japanese movie The Admiral there is more background. Nagumo was under strict orders to return all of the carriers safely to Japan. Since the American carriers were not at Pearl Harbor he worried that they were in the area and could attack at any moment. He was following orders.
Unfortunately for Japan and fortunately for the U.S., Admiral Nagumo was the wrong person for the job, he was promoted to command of the Carrier Strike Force due to his seniority. He was competent but no experience with carriers (few at that time did have it). He did have enough leeway is his orders to take advantage of the situation, but was opposed to the attack in the first place. If the third strike had been launched, the U.S.S. Enterprise would have most likely been found and sunk.
He had reports that our submarines were in the area. This was his biggest concern.