Also remember that the next door neighbor is giving Walt the stink-eye in that first episode (509?). In the next episode they show her talking to Walt and she’s very nice to him, so somehow between then and “now” when Walt comes back to get the ricin, the negative information about Walt has made its way into the community, at least enough for the next door neighbor to know about it.
The next door neighbor probably also noticed the house next door being fenced off