The davidovits, josephdavidovits, geopolymer, and geopolymerization keywords, sorted, duplicates out:
Well, if true, they needed to make cement which means they had to kiln limestone and clay. Lot of work. Where did they all the fuel to do the kilning from...I know! The Sahara Forest!
I think that it would actually require more steps and work to create concrete than quarry the rock, not that working with stone is particularly easy.