They lived in the outlying area of Wroclawek, which is northwest of Warsaw. They were shoved in a farmhouse with two other families. He often told of having to sneak out to find wild fowl and firewood. They weren't Jewish and I am not sure the whether American passports had anything to do with the Germans pretty much leaving them be but it must have helped in some way. He was only five when it happened so his memory wasn't all that good about what happened. Also, I get the feeling he simply didn't like to talk about it.
Did they choose to stay in Poland rather than go back to the U.S.? Wouldn't the Germans would have allowed them passage because of their U.S. passports? Germany and the U.S. weren't at war until Dec. 41.
They were fortunate they weren’t stuck on the Soviet zone of occupation in 1939, the fate for Poles there was even worse than under Nazi occupation.