Cutting perfect concentric deep-groove circles by hand? Sure. Easy-peasy.
Once the center pilot has entered the work, fly cutters would be well behaved. The fly cutter would be mounted so as to engage the work after the central drill was seated. Look closely. That is exactly what you will see.