They launched their aircraft from a point only 200 miles North of Oahu (2). After worrying that the American carriers weren't there and the potential of American submarines, the Japanese carriers went bake to Japan (3).

In the Japanese movie The Admiral there is more background. Nagumo was under strict orders to return all of the carriers safely to Japan. Since the American carriers were not at Pearl Harbor he worried that they were in the area and could attack at any moment. He was following orders.