Those who knew cloth would not have used the terms interchangeably. The costs were considerably different. Byssus today is over a $1000 a square foot! It was 100 times more costly than silk in medieval times, so a knowledgeable person would not confabulate silk, cambria, and Byssus! Byssus was essentially reserved for royalty.
The only source we have for the inability for paint to stick to Byssus, is the modern weaver lady... And frankly I doubt her. Raphael wrote about getting his best result on Byssus. . . with paint. If you can dye it, you can paint it. There is nothing magical about Byssus that would make it shed paint any more than any other organic substance. One can paint a pearl... One reason that few examples of Byssus cloth survive is that moths love to eat it! It is very fragile.
when I was researching the Manoppello Veronica in 2006, my recollection is that the Raphael/Dürer letters were quoted in an art book published in 1936 or so. . . I cannot comment on the accuracy of the translations as I have no expertise on sixteenth century German. I relied on the translations. However the art book was not discussing anything about Veronica's or the Manoppello veil, just the unique attempt at transparent art the two famous artists were attempting, with varying degrees of success. Raphael was not too pleased with the difficulty of working on the diaphonous cloth and it's tendency to stretch, making accuracy difficult. The translations had been actually done in he nineteenth century, IIRC.
Some people think the Manoppello veil is Dürer's self-portrait... But it just doesn't match his looks. Here is his own self-portrait from 1500:
Albrecht Dürer c. 1500
You can see his hair, beard, and mustache are much more pronounced than the image on the Manoppello image. He just does not look like it.