The fact that a right wasn't enumerated in the Constitution doesn't mean it didn't exist. There was tremendous opposition to writing the Bill of Rights, because the founders believed such a bill would be used to LIMIT rights rather than guarantee them. The 9th Amendment was added to assure that the people had many rights besides those enumerated.
So they aren't growing. They are being identified by the courts.
The problem I have with a right to privacy is that it is being used to override more essential rights: those of life and liberty. When your unenumerated right infringes upon my enumerated right, then we have a fundamental violation.
My point is some would consider the rights to be growing by the justices identifying them.