The other claim is that such hyper-velocity leads to rather rapid barrel wear. Since varmint hunts tend to burn up a lot more rounds than big game hunts, this could be a concern.
I personally can't vouch for either situation, as I have never owned one, (would very much like to someday though). For now, I am satisfied with my .22-250 at 3850 fps. and short bolt throw, (the Swift is a long ol' cartridge). Hope this helps. Maybe someone who has more experience with the Swift can chime in here.