Gee, it seems to me that it should be easy to prove, or disprove, your theory - simply by examining the limestone blocks to see if they are composed of rubble cemented together.
Do you suppose that any archeologists have ever bothered to do that?
Regards,
That also explains the ungodly tight fits of the blocks, it’s what you’d EXPECT from pouring the things.