Start with an electromagnet in the OFF position.
Magnetic field covers no area.
Now turn it ON.
Magnetic field now covers some finite area.
How did it go from covering no area to covering a finite area if it did not propagate outwards from the electromagnet at some speed, per your claim above?
Now, take an electrically charged object and measure the field at some distance. Then move it to one side, very quickly. How long will it take for the change in the field to register at your apparatus?