I in no way mean to add to your guilt, Friend, but you should have called 9-1-1. You witnessed assault, and should have intervened.
That's exactly the lesson I learned, and it is what I should have done at the time. Last summer I called 9-11 when a different neighbor left her kids--the oldest was four--alone in her apartment for three hours. (The mother didn't come home until the next day.) The police arrived within five minutes of my calling and they immediately summoned CPS. There was no food in the house and all three kids were in diapers that were probably a day or two old. While the officers were waiting for CPS to arrive, the kids' aunt (who also lived in the apartment) showed up and basically said, "Its about time you people did something," as if she'd called for help and got none. She knew what was going on in the house and never did anything about it. To the best of my knowledge, the mother never tried to get custody of her children back.