OK, I'll try to, but I think a cosmologist or an origins theorist would do a better job than I.
A "steady state" theory of cosmology posits that the universe is infinite in age and in a steady state. That it always was, and always will be. Infinite time makes it very easy to explain away the need for a transcendent creator: because given enough time, even the most "improbable" of events, such as the spontaneous generation of life and its evolution to human intelligence, are hardly improbable at all: given enough time, they are certain to happen.
A similar argument can be made for "oscillation" theories about our universe: if our big bang was just one of an infinite number, then we humans happen to be "lucky" enough to exist during one of those oscillations where the conditions were just right for our spontaneous creation.
That's why the rapidlly coalescing consensus that our universe is finite in size and only 15 billion years old---and that it could not possibly be oscillating, because its expansion is accelerating, not decelerating---causes a genuine problem. First of all, it causes a mathematical problem, because improbable events like the spontaneous generation of life now are genuinely improbable because they have a finite opportunity to occur.
And secondly, it causes a philospohical problem, in that because our universe has a beginning, that it should somehow have a cause. Something that is infinite in duration does not need a cause; something that is not, does. Admittedly I may not be defending that point very well; you may ask, "why must it have a cause just because it has a finite beginning?" Like I said, that's why I say that cosmologists would probably give you a better answer than I.
But even though I'm not defending their position very well, the fact remains that they agree that the finite beginning of this universe is just "not enough" to provide a complete explanation of the origin of this universe. Somehow, there needs to be something more. Thus, cosmologists come up with ideas that connect our finite universe to something infinite and uncaused. One common example is the multiverse theory: our universe is but one of an infinite number of alternate universes, and this one just happens to be the one that produced our existence. This is another way to provide the "infinite" opportunity required for improbable events like the spontaneous generation of life to occur with certainty. After all, if there an infinite number of universes, and they cover all the possible ways that the universe could have evolved, the certainly at least one would turn out the way ours has. Some also hold out hope that perhaps our universe will be proven to be decelerating, eventually moving to a collapsing "heat death." Then they could argue some sort of circular timeline with no beginning or end. Alas, evidence of a "cold death" to this universe seems to be mounting.
So I'm not saying that an atheist must believe in infinity. But the ones who study the universe's origins most certainly seem to.
For one thing, for atheists to theorize and make issue out of the "probability" of the creation of the universe raises many questions. For one, it raises the question of how such an event could be considered probabilistically. Is it just a conceptual model for uncertainty in the way people usually apply probability, or does it refer to some truly random feature of the physical world, akin to conventional thinking about quantum physics? If the former, what is the mechanism? Then, even if probability could be applied to whether the universe came to exist or not, then of course it would be a nonissue, since if it did not happen, we wouldn't be here to discuss it. That is, however unlikely, if it happens, then here we are.
Then of course for atheists to talk about infinite time leads to a whole problem of identity. The only coherent concept of infinity that I can think of comes from mathematics. But that does not apply to physically observable quantities such as length or time. For atheists to base their metaphysics on the existence of a literally unimaginable, and as far as I can tell totally incoherent, notion such as infinite time rips them as far from reality as any of the self-proclaimed mystics.
Finally, atheists apparently not only put themselves at odds with theists, but also with contemporary cosmologists. You would think that a theory of finite time, which makes the notion of a creator of time absurd, would be acceptable to the atheistic view.
We are fascinating creatures in our flaws. Atheists claiming not to be mystics, like Democrats claiming to help people, are just two examples.