Yes, the fob is secure because it uses PKI and the private key cannot be stolen from the fob. Ewerything else you use, from passwords to questions to text messages to any kind of software can be compromised and under the right conditions compromized en masse. Not just one vote, millions of votes if 2FA is your voting "security"
You won't get notified by email either or if on the odd chance you do I'll make sure you get dozens of notifications so you will ignore the useful one (actually Google does that already when I use the same account on multiple laptops and phones).
If you really want a purely tech solution then hand each voter a PKI fob in person upon presentation of ID and credentials. I've written browser extensions, both PKCS #11 and Microsoft CSP. Software tokens are not secure, only hardware is secure. I've also written the server side java and client javaascript to use FIDO U2F to secure web accounts. FIDO U2F fobs are secure just like the PKI fobs (some fobs do both).
The server side software would be open source, you WOULD be notified by email if your account was changed (that is very standard practice on my planet, I don’t know about yours), and the most important thing is that any large-scale tampering would be NOTICED. People would look up their ballots in the published list and find they don’t match, and red flags would go up. As for the tabulation, anyone would be able to download the published list into Excel and run the totals for themselves.
Transparency means you can see the evidence of tampering easily. The only thing an attacker could do is spoil the election possibly, but there would be no way to alter the outcome without detection.