You made the right call. I’ve done the same. It amazes me sometimes how stand-bys or other late flying cheap ticket holders want someone else who specifically planned to keep out of a middle seat to accommodate them because they are ‘family.’
They’re on the same plane for a trip that doesn’t last days and weeks where they’ll never see each other.
Kids or no kids, I’m not sitting in a middle seat. To your point, I’ve even paid more for the isle seat so yeah, not giving it up.