But time dilation still applies. Consider the twins problem (one twin leaves on January 1st, travels close to the speed of light, and returns on December 31st) where the travelling twin carries the far end of the wormhole along for the ride. A year elapses for the one end of the wormhole, but only a week or so has elapsed for the "far" end (which has been brought conveniently back home). The well-travelled end doesn't just connect to the end that stayed on Earth; it also connects to the previous January 8th. That "little step" now takes you 51 weeks backwards in time, instead of across the galaxy.
An excellent point, and one that I have some trouble visualizing myself!