The server side software would be open source, you WOULD be notified by email if your account was changed (that is very standard practice on my planet, I don’t know about yours), and the most important thing is that any large-scale tampering would be NOTICED. People would look up their ballots in the published list and find they don’t match, and red flags would go up. As for the tabulation, anyone would be able to download the published list into Excel and run the totals for themselves.
Transparency means you can see the evidence of tampering easily. The only thing an attacker could do is spoil the election possibly, but there would be no way to alter the outcome without detection.
As for your voting system, as I told the OP, there's a lot more to it. Once you solve the registration problem the voting problem is pretty simple. Also I should not have said in my last post that all you need is the fob. If that's the case, then you could sell your fob and thus sell your vote. Otherwise I think fobs are great and I am a fan of them.
Another way to sell your vote is the lookup system you described.