Bad argument I am afraid. Islam is not a unified organisation. There are LOTS of sects of Islam, some of which are violent, some of which are not. A better analogy would be if you joined an environmental group that wanted to (for example) save the Panda by not lobbying government to ban the killing of Pandas. Meanwhile another, seperate group who also wanted to save the Panda by killing people who shot pandas. You would not be responsible for the actions of the murderers, would you?
Then the non-violent sects you claim should stand up and clearly condemn the Islamofascists. Except for the recent protests in Jordan, this has not happened, except on a very small, practically onesie-twosie scale.
The few individuals (Muslim and otherwise) who do speak up and condemn terrorism as an illegitimate expression of Islam are themselves condemned.