The Vatican Museum states that the man to the right of Marcantonio Raimondi, who others have thought is Durer - is Raphael. And that does look like the Veil man. I assume the description means the viewer’s right and not the man in the picture’s right.
Apparently, the view that it is Raphael is a fairly new view as remarked by one author as being a 20th century tradition. Not sure when this idea first came up. Raphael’s patron, Julius II , is the one being carried.
Julius II would obviously have access to byssus as we use the term today.