Yes, it's all a matter of degrees ... having said that, it would take an awful lot of 'matter' hitting the magnetosphere and ionosphere to have an effect. We are more likely to see the earth's magnetic field change due to internal forces (in fact, it moves yearly, in fact, it moves daily) ...
Daily Movement of the North Magnetic Pole (Including diagram)
That I remember from Boy Scout days. Thanks.