Although Elrond and Galadriel both imply that this "fading" is already underway, with a strong implication that it has been stopped or slowed during the Third Age by the power of the Three Rings (which the Elves can use as long as Sauron doesn't hold the One). With the destruction of the One Ring, the Three also lose their powers, so the fading will accelerate greatly.
Why doesn't Saruman steal his Ring from Gandalf when he is holding him prisoner? Isn't he aware that Gandalf holds one of the Three? Wouldn't Gandalf have told his superior sometime over the previous thousand years? Or is he unable to take it away against Gandalf's will?
BTW, how about that auto-homing staff of Gandalf? Saruman takes it away, and he doesn't have it when he escapes on eagle back. But then he has it again in Rivendell.
He presumably loses it again on the Bridge, but then he has it again (also presumably his sword Orcrist) in TTT.