I almost feel that you must be a neighbor. We, too, own a Victorian house (a cottage, really) that was long neglected. Unfortunately, it’s still a tiny bit neglected but much better than when we first bought it. Our neighbors are complete slobs (yard a mess!)but the nicest people on earth. We really love them. I learned the hard way as a condo owner at one time, that leaving people alone is the only way to be a good neighbor.
You take a shot, win or lose.
I have the same type of neighbors as you describe, right across the street.
They had an old house, in great disrepair when we bought our place, but through the years they've improved it as well as they could and done a pretty good job. During this period, they also lost a daughter to a car accident and raised her very young daughter when they were in their 60's and the he was severely disabled.
They try to do the best they can. I trust them to keep a watch on my property and they do the same.
I knew what I was buying into.
Hell, if you ever move into Maine, I'd be thrilled to have you as a neighbor.