The Roman Catholic historian Malachi Martin attempts to confine these lines of desposyni as follows. These were:one from Joachim and Anna, Jesus’ maternal grand parents. One from Elizabeth, first cousin of Jesus’ mother, Mary, and Elizabeth’s husband Zachary. And one from Cleophas and his wife who was also a first cousin of Mary (M Martin Decline and Fall of the Roman Church, Secker and Warburg, London, 1981, p. 42).
He acknowledges that there were numerous blood descendants of Joseph (p. 43) but, as all Roman Catholics, he seems to attempt to deny their direct lineage from Mariam or Mary, even though he acknowledges they had clung to the Church throughout the early years. Maria is removed to first cousin and not sister as the Bible says.
Martin records that the descendants, as leaders of the Church, held a meeting with Sylvester bishop of Rome about the whole nature of the Church in the year 318 CE (ibid.). The emperor provided sea transport as far as Ostia for eight of them and then they rode on donkeys to Rome and the Lateran where Sylvester now lived in splendour. They wore rough woollen clothes, with leather boots and hats. The conversation was in Greek as they spoke Aramaic and had no Latin, and Sylvester spoke no Aramaic. Martin considers it probable that Joses the oldest of the Christian Jews spoke on their behalf. (CCG, "The Virgin Mariam and the Family of Jesus Christ")
The Roman church had come to be wealthy under patronage. When Constantine tried to establish the Christian system in order to use it, he gave the edict of Toleration at Milan circa 314 CE. After this, the Christians became influential. In 318 CE, the emperor paid for the travel of the family of Christ to Rome to confer with bishop Sylvester at the Lateran Palace. He was, by this time, a very wealthy person living like a prince. The party of the family of Christ came by ship to the port of Ostia and then they went by donkey to Rome. They were dressed in woollen homespun garments and leather hats and boots. They spoke Aramaic and Greek. Bishop Sylvester spoke only Latin and Greek so the conversation was carried on in Greek. It seems likely that Simon was their spokesman (see the paper The Nicolaitans (No. 202) and also the ex-Jesuit historian Malachi Martin The Decline and Fall of the Roman Church, pp. 42 ff).They expressed their concern that the laws of God had been removed as the basis of the church. They wanted the Sabbath reinstated. It had been made inferior to Sunday from the Council of Elvira in 300 CE. They argued for the Torah, which was the Hebrew name given to the law of God, to be reinstated to its correct position. That included the Holy Days and food laws. They asked that the Greek bishops put into Alexandria and Antioch and elsewhere be replaced by the family of Christ. They asked also that Jerusalem again be made the centre of the faith and the money for the church be able to be sent there.
They went home. Instead of using his influence with Constantine to reform the church of these Gnostic influences and restore it to the true faith once delivered to the saints, as the brother of Christ wrote (Jude 3), Sylvester set about destroying the faith and the family of Christ with it. (CCG, "Modern Christian Fundamentalism: A Contradiction in Terms")
I found the actual quote:
...A meeting between Sylvester and the Jewish Christian leaders took place in 318....The vital interview was not, as far as we know, recorded, but the issues were very well known, and it is probable the Joses, the oldest of the Christian Jews , spoke on behalf of the desposyni and the rest.
...That most hallowed name, desposyni, had been respected by all believers in the first century and a half of Christian history. The word literally meant, in Greek, "belonging to the Lord." It was reserved uniquely for Jesus' blood relatives. Every part of the ancient Jewish Christian church had always been governed by a desposynos, and each of them carried one of the names traditional in Jesus' family---Zachary, Joseph, John, James, Joses, Simeon, Matthias, and so on. But no one was ever called Jesus. Neither Sylvester nor any of the thirty-two popes before him, nor those succeeding him, ever emphasized that there were at least three well-known and authentic lines of legitimate blood descent from Jesus' own family...
...The Desposyni demanded that Sylvester, who now had Roman patronage, revoke his confirmation of the authority of the Greek Christian bishops at Jerusalem, in Antioch, in Ephesus, and in Alexandria, and to name desposynos bishops to take their place. They asked that the practice of sending cash to Jerusalem as the mother church be resumed... These blood relatives of Christ demanded the reintroduction of the Law, which included the Sabbath and the Holy Day system of Feasts and New Moons of the Bible. Sylvester dismissed their claims and said that, from now on, the mother church was in Rome and he insisted they accept the Greek bishops to lead them.
...This was the last known dialogue with the Sabbath-keeping church in the east led by the disciples who were descended from blood relatives of Jesus the Messiah. [Decline and Fall of the Roman Church, New York: Bantam, 1983. pages 30-31]