Actually, it's the fact that your eyes move around the picture that creates the illusion of motion. They would also appear to move if they were in black and white. When you fix your vision in one spot, they stop moving.
As I typed that, what you wrote is what I was actually thinking. Color blindness can keep one from seeing the contrasting parts that “move”.