But he apparently is not simply passing by. Why else would there be the mention of “he will not see me stopping here”? He is obviously in a place that is not usual. Thus, the symbolism.
“Why does a poem have to be symbolic?”
It definitely does not. An example is the many, many lyrics of music, especially popular music, much of which is ‘poetic’ because of its patterns.
The application of symbolism is the choice of the individual who reads, including the poet himself/herself who creates it.
Just as when one reads a story in the Bible the story may resonate with the reader on an individual level (and the very popular concept of small groups today, may cause individuals to share their experiences/reactions to various passages in the Bible) poetry is the same way; a different result/experience for each individual, including the individual who enjoys simply reading the words and picturing the scenes described.
As to Frost’s crafting; he was a wordsmith and very fine word craftsman so perhaps his poems sometimes drew him to work and rework them, and perhaps sometimes they flowed easily and did not entice him to rework them.
Fine poetry and writing usually does draw the writer in to working and reworking because the playing with words is to a writer the joy of the craft.