I'd love to get a lawyer's take on this, but I don't believe you can do this after you've already submitted a sworn statement on the matter.
I'm not a lawyer, but I remember the 5th being used many times in Senate testimony after whispering between lawyers/client. As a matter of fact, I believe the only exception to the 5th is in Grand Jury testimony which is only used to decide if there is enough evidence to proceed to indictment.