I had a similar experience years ago traveling from Europe to the Middle East where we lived. I had a 2 yr old and a baby with me. Middle of the night..people sleeping..suddenly the plane just drops. Screams, cries, everyone asking what happened..and then it happened again. Everyone was terrified. The pilot never said a thing. Later we were told we were above Turkey where the wind drifts over the mountains and valleys were especially bad. Flying lost all its magic for me after that.
A commercial pilot friend told me after our incident that this CAT (Clear Air Turbulence) was not dangerous but very unnerving for the passengers. I agree with you that the incident we were in took the fun out of flying for me too.