How much RAM do you have installed on your Mac? The proper answer is probably not enough. . . but seriously, how much?
I have the maximum possible for my late 2013 iMac, 16 GB.
I’ve used Activity Monitor and iStat Pro to check, and when Safari freezes there has been 6 - 8 GB of memory free.