"Hoist" was in Shakespeare's time the past participles of a verb "to hoise", which meant what "to hoist" does now: to lift. A petard (see under "peter out" for the etymology) was an explosive charge detonated by a slowly burning fuse. If the petard went off prematurely, then the sapper (military engineer; Shakespeare's "enginer") who planted it would be hurled into the air by the explosion. (Compare "up" in "to blow up".) A modern rendition might be: "It's fun to see the engineer blown up with his own bomb."
Is there a particular book you found that in? The explanation of the Shakespeare quote I mean.
Dictionary goes on to say that “hoist with one’s own petard” means ... “destroyed by the very devices with which one meant to destroy others.”