Sadly, the truth is his mother is dead. She died in a car accident around Christmas when he was little. He and Hunter survived it but his mother and their sister did not. Ole Joe likes to trot out that story to get sympathy for himself but if you step out of politics for a moment, you'll acknowledge it was a terrible tragedy (despite the fact that the accident was her fault and the family publicly blamed the truck driver for many, many years).
Joe was left to raise the 2 boys alone and didn't do a very good job in my opinion.