I don't know about the other buildings (I haven't been in London that often) but Clarence House is simply the official residence of the Prince of Wales, his wife the Duchess of Cornwall, and both his sons, Princes William and Harry. It's been in the family since it was built by William IV, who at the time was Duke of Clarence.
It is common practice to refer to the Prince of Wales' private office as 'Clarence House' in the sense of saying for example: 'Clarence House issued a statement' - similar to how American media use the term 'the White House' to refer to the administration of a President.
thanks!