I don't think of it as a physical death but a physical change - They are in their spiritual bodies. Those that have passed on before us are in that body now and when Christ returns:
1 Corinthians 15:52 In a moment, in the twinkling of an eye, at the last trump: for the trumpet shall sound, and the dead shall be raised incorruptible, and we shall be changed.
53.For this corruptible must put on incorruption, and this mortal must put on immortality.
At that time we will all be in an incorruptible body (for the millennium) - no one has died yet, there was just a change. Our bodies are gone but no one is dead. Some, at that time, have also become immortal. Those are the ones who take part in the first resurrection (Rev.20:4-6). The others, who are in an incorruptible body but are still liable to die (not immortal) must wait until the second resurrection. They are considered the "spiritually" dead (Rev.20:12) and are judged according to their works during the millennium. They aren't judged on their belief in Christ as they can see Him. They know there is a Christ.
So.....When our spirit leaves our bodies we change. Some become immortal but others are considered spiritually dead - that is the first death. The second death is that of their soul:
Matthew 10:28 And fear not them which kill the body, but are not able to kill the soul: but rather fear Him Which is able to destroy both soul and body in hell.
That is what I believe those scriptures mean. Another meaning has been explained to me and it sounds plausible but I don't understand it well enough to explain it. I don't know if anyone will ever be certain until we are there but I do know - I want to be part of the first resurrection.