A ray also has infinite length -- but also a starting point.
Think of timehas height above the center of the earth.
(i.e. the question "what is below the center of the earth?" is nonsensical.)
If time has a point of origin something must have originated that origin.