The context was as bad, if not worse, than the single statement, showing scorn for hard work and intelligence and praising government involvement instead.
But that’s beside the point. Even if we allow “you didn’t build that” as out of context, it resonates because it still tells the fundamental truth of what he actually believes and practices.
Sorry, but he can't get away with an after the fact out-of-context claim. After talking about "roads and bridges" he then said if you own a business, you didn't build THAT. If he was referring to not building roads and bridges, he would have said: "If you own a business, you didn't build THEM" (referring to the roads and bridges). He didn't, he said you didn't built THAT (referring to the business)
He only regrets saying it after it's a problem for him, if he would've said he regretted it on his own it might have been more sincere but sincerity is above his pay grade.