It could be either. But does it really matter? Dylan left it up to the listener to unravel the mystery, and that's just fine with me. A lot of his best songs were that way.
I love the Byrds' version of the song, but it's a shame they never (to the best of my knowledge) recorded the complete version as Dylan wrote it.
Best version of Mr. Tambourine Man is from the Concert for Bangaladesh. HUGE!
You're right, it doesn't matter, and good poetry is more than just descriptive, drawing the listener into the experience.
I prefer the Dylan version. Muscially, the Byrds' is far superior, of course, but Dylan's - listening to it I can picture him at daybreak, standing alone, reflecting on the things he just saw, heard, and experienced.
If you rate the excellence of poetry by the ability to capture and communicate an experience then I've never heard better. Every line is a treasure.