Hmmm. I'm not sure your logic holds up; ironically, it might be the very direction of time that monkey-wrenches the works.
What I mean is, probability becomes meaningless when you look backward in time; it only has meaning when you look forward. Let's say I buy a raffle ticket, and I win. Before my winning ticket was drawn, my odds of winning were very-large to one, against...yet the odds of someone winning the lottery were one-to-one, or unity. Somebody will win.
So when I go to collect my raffle prize, the judge looks at me and says "The odds of you winning instead of all those other entrants are so small, it's highly improbable that you in particular would pull the winning ticket. You must have rigged the game somehow!"
No I didn't. Now that I've won the raffle, my odds of having drawn the winning ticket are one-to-one. It's already been done.
I was just speaking of Carroll's use of the word 'unnatural'. The low-entropy beginning of our cosmos does appear to be very highly improbable...which is not to say it can't happen. Indeed, given a long enough time, just about anything will happen (there are no observers waiting around for most of the time so nobody's counting the hours!).
That 'somebody' is ALWAYS the originator of the 'lottery'.
It is NEVER a given that someone will choose a 'winning arrangement of random numbers'.