I always thought the implication was that he'd fought for Carthage. When I was a kid, Hannibal was a hero of mine. But now, I tend to admire the Romans more for coming back from several horrific defeats at Hannibal's hands. Winning the Punic Wars really established Rome as a great empire.
I’ll have to re-watch the scene again, as you raise an interesting point as that he may have been a Cartheginian as opposed to a Roman soldier.
It was funny, though, how he spoke to Omar Bradley, saying “you don’t believe me Omar, do you?”.