Try telling that to Harry Daghlian and Louis Slotin. The exact same plutonium core killed the both of them in separate incidents without exploding or either of them breathing it in.
.
If one spends too much time in the presence of that crap it can get you.
But we shot the pile of plutonium for quantities, which took about 4 hours, frequently and never had any ill effects. (back in the ‘80s)