Sure you can. 🙂 Category 1 and 2 poisons haven’t earned the skull and crossbones for no reason.
Taber's Medical Dictionary - PoisonAny substance taken into the body by ingestion, inhalation, injection, or absorption that interferes with normal physiology.
Virtually any substance can be poisonous if consumed in sufficient quantity; therefore poison more often implies an excessive degree of dosage rather than a specific group of substances. Aspirin is not usually thought of as a poison, but overdoses of this drug kill more children accidentally each year than any of the traditional poisons.
medical-dictionary.thefreedictionary.com/poison
a substance that, on ingestion, inhalation, absorption, application, injection, or development within the body, in relatively small amounts, may cause structural or functional disturbance. Called also toxin and venom. adj., adj poisonous.
Synonyms for 'poison' toxin, venom, bane (archaic)
I guarantee you that no doctor would agree with your definition.