That's good enough for me and sounds like he retracted what most people took for the statement's meaning...my own feeling is that you should let it go now after those two comments he made, otherwise, you begin to look to be about as intolerant as you claim he is.
Just my opinion...it cost nothing and you can take it for what its worth.
The Left is far too fond of such comparisons to begin with, calling conservatives fascists and such. Let's not give them any ammo to make it look like they actually have a point.