It my understanding that the original writers created the Doctor's "regeneration" in an act of desperation when the actor who played the first doctor retired unexpectedly in the middle of a season because of deteriorating health. The public bought it, and it's been used ever since. I remember watching the show back in the late 1970's when Tom Baker was the doctor, it was on PBS about midnight during their "British Programming Block" which consisted of Benny Hill, Monty Python and Dr. Who (Until I was a teenager I thought those where the only three shows on British TV!)
Its nice to have a great sci fi show without wildly overdone special effects.