Actually I consider the empty box a sign that it once did carry the bones of James, the half-brother of Jesus. The veneration of relics is a heresy that didn't pop up until later. It's definitely an anti-Scriptural practice. Once the heresy of veneration started, people started emptying the ossuaries of dead apostles and prominent early church members to add to their collections. I would imagine that the ossuary contained bones until around the mid-2nd century.
Veneration of relics is not a heresy. And it is not anti-Scriptural. BTW, the veneration of relics is recorded in the earliest Christian texts (earlier than the Church established canon of New Testament) like for example martyrdom of Saint Policarp - from 86 AD.
You want Scriptural evidence of miracle-working holy relics of a saint? Here you are:
And Elisha died, and they buried him. And the bands of the Moabites invaded the land at the coming in of the year.
And it came to pass, as they were burying a man, that, behold, they spied a band of men; and they cast the man into the sepulchre of Elisha: and when the man was let down, and touched the bones of Elisha, he revived, and stood up on his feet.
Why the relics of saints are holy and miraculous? Becasue they are belong to bodies of living holy men who are standing before the throne of Creator.
You are confusing the post-scholastic bookish speculations of Luther and Calvin with the actual early Church.