If you physically have the gold, where do you store it so it is safe? If you do not have physical possession, who do you trust with keeping it safe?
I am not a sophisticated investor so I believe I am vulnerable to con men when it comes to gold. Perhaps you found an honest broker.
Now let us say the worse thing comes about, the economy collapses. you are sitting pretty on all your gold. Where are you going to spend it? If you find someone to take it in trade, how long will it be before everyone and their brother knows you have a stockpile of gold? How many are you prepared to kill to protect your gold?
A condo is not a house. Here not only does the government “own” your condo, but so does the condo association. Perhaps for you that was a bad decision. However that does not mean everyone that paid off their home has made a bad decision.
I wish you luck with your gold. I believe it gives you comfort today but gold is a risk the same as any investment even if you don't admit it to yourself.