I think he did say it. My issue is what he says and how he says it. I don’t think the definition he stated there of what constitutes a ‘conservative’ is all that compelling.
William F. Buckley summed up the role of conservatives with his classic line “standing athwart history yelling stop”. So Sowell isn’t exactly off the reservation with his comment.
But again, what’s most interesting about Sowell’s remarks isn’t so much what he said about conservatism, but that he selected the label libertarian as the best match for him. Presumably this was because libertarian core principles are closest to his own, and even though yes he does note that he disagrees with libertarianism on some issues. I do think it’s telling that he apparently agrees with libertarians on the drug issue, which for many conservatives is the bright line between conservatism and libertarianism.