That quote is quite revealing. Their behavior was such that the next door neighbors wondered what *they* were doing, not what *he* was doing.
I thought that quote was revealing/disturbing too. I've never wondered about neighbors "What do they do over there?" unless I suspected something weird or disturbing.
Remember this oldie?
(Midnight, one more night without sleepin')
(Watchin' till the mornin' comes creepin')
(Green door, what's that secret you're keepin?)
There's an old piano And they play it hot behind the green door
Don't know what they're doin' But they laugh a lot behind the green door
Wish they'd let me in So I could find out what's behind the green door
(Knocked once, tried to tell them I'd been there)
(Door slammed, hospitality's thin there)
(Wonder just what's goin' on in there) . . . . .