You also cannot, after claiming the 5th and making a statement, answer a question from the chair and then just clam up.
For the sake of arguement, why not? Is the 5th Amendment conditional, or absolute? You either have a right to not self-incriminate, or you don't. Afterall, answering some questions can't lead to self-incrimination, while others might.