The ones that got torched were the “stragglers” the main force had already delivered the gold to the castle.
I normally watch each episode twice to pick up details I missed. This is one of those details. Is there dialog in the episode to show this? The only part I remember about gold is when Jamie gave a sack of it to Bronn and Cersi telling the man from the Iron Bank that it was on its way.