Oh, I understand completely. That’s the fudge factor. What you may not be taking into account is the original sin of Adam. Calvinists say that was also predestined. So using the term “passing by” someone now doesn’t absolve them of anything. They still confess that the entire “chain of events” was predestined. If God predestined Adam to sin and all have that sin nature because of it there is no getting around that it’s double predestination.
I’m not sure we have the same understanding. I believe double predestination to be that God predetermines both the saved and the damned. He picks the saved, and He picks the lost.