How hard is it to lose your phone?
That doesn't matter. Only you can use ApplePay. First of all only the owner of the iPhone can open the iPhone. . . or use ApplePay because it requires the owner to use their living finger, not a mere copy of a fingerprint, to activate it. Merely losing your phone, or having a thief steal it, will not give the finder or the thief access to ApplePay. It is secure. That's why the major credit card and banks have accepted it and the liability that goes along with using it. They think they will be saving HUGE amounts from the decrease in credit and debit card fraud with ApplePay.
In addition, once you know you've lost or had your iPhone stolen, you can deactivate your iPhone remotely from any computer, turning it into a useless brick for anyone except its true owner, you. Only you can restore it to functional use and that requires your AppleID. That STILL won't get them access to ApplePay.
Sorry, your point is not valid.