Of course he had a specific appearance. Isn’t that a given? It has nothing to do with our faith though.
It has nothing to do with our faith in what? That He was true God AND true Man? The latter demands, as you acknowledge, that He had a specific appearance.
Perhaps (just maybe, IMHO) He left the shroud as a sign to a materialistic and unbelieving age that "Yes indeed, folks, I REALLY WAS HERE".