Sometimes it’s an adverb. Sometimes it’s a prepositional phrase or even a clause!
You’ll notice that I cleverly split the infinitive in my question ...