You thought wrong. Most ejections from fighters allow the pilot about 2 seconds of decision making time at best. The reason that there are few incidents of fighters contacting neighborhoods is that the approach patterns to most bases are undeveloped.
If a pilot has time and control of his aircraft, he/she will try to LAND it, not just steer it away from homes. When a fighter decides that it aint flyin anymore, it rarely allows you to steer.
An old friend of mine, one of those guys with an uneven T/O - Landing count, had to punch out of af F-100 north of Madrid around 1960. I asked him what he thought when he realized the A/C had died.
His answer; "Throttle, Bottle, Left, Right, Squeeze".