The friction from the tides of the moon and sun, the atmosphere, and the internal magma should add up to more than a millisecond every hundred years as is now observed.
I think you've underestimated it. As I understand it, the earth's rotation is slowing approximately .005 seconds per year, which would add up to half a second per century. That still doesn't present a problem, even extrapolating back 4.6 billion years. I'm not at all sure what you're trying to argue.