You are not very logical. The freedom of contract can lead to the contractual obligations limiting freedom.
In order for transactions meeting the requirements of the theory of perfect competition the parties must be free to make the choices. A gunman threatening to shoot you if you do not turn over your wallet means that when you do so it was not a free choice. Nor is it acceptable to say "well I had the choice of surrendering it or getting shot."