Yah, it's either the starter or the battery, and I don't have the experience to say which is more likely. OTOH, the rate of progression would seem to implicate the starter.
I assumed you knew enough to be able to eliminate the battery as the cause. It sounds like the starter is "dragging."
Don't wait for it to fail altogether. That could be ... bad.