I believe that would give them a crumb through which to find your wallet and then they can track your purchases
assuming they can crack the encryption (which has me wondering these days after the snowden / SSL revelations)
Good point. I doubt they can crack the encryption, but wouldn’t doubt they already know many of the addresses.