The alternative is to limit unreasonably what parents and children can do. No, you don't "want" speculation, but you can't control what other people think, nor what they say, if they're the kind of people who are going to gossip and speculate. Being that kind of person is its own punishment.
It sounds to me like the young woman and her father had a great experience that they can both treasure the rest of their lives - perhaps a reward for her high school graduation, or earning a college scholarship?
I understand that. I asked those neighbors what they knew that gave them reason for doubt. They couldn't give me any reason. I told them they should give the benefit of the doubt instead of forming a derogatory opinion. Frankly, I doubt they agreed or appreciated being told.
My point is, as you say, we can't live our lives based on what others may think, but we should, perhaps, consider that our actions, however well meant, may be open for interpretation.
I myself have taken my kids on camping trips (as did your dad with you). However, I'm not sure I'd ever opt for a two week cross-country trip with one daughter while others are relegated to home. First, it could (and obviously did) appear unseemly to some, but more important, it ignores the rest of the family. There's other ways to give deserved time to a child than such extended exclusive attention. Why shouldn't the whole family enjoy such a marvelous trip? If anything relating to that one daughter was intended, that could be attended in the course of the vacation, with them along as well.
Then there's the fathers resultant inattention to that daughter's two siblings left at home. Those are the problems I have with the situation.