Am I sort of right about Bitcoin being tied to prime numbers?
You are.
Bitcoin depends heavily on public-key cryptography, which in turn depends on the difficulty of obtaining the prime factorization of extremely large numbers (way bigger than 2**20).