"We're not talking about what is philosophically valid, but what is scientifically valid. How does science determine this distinction?"
It doesn't. Methodological naturalism is the philosophical viewpoint science has taken to ensure objective research.
"Is it the understanding of the observer that renders a phenomenon 'natural' instead of 'supernatural?'"
Oh certainly, an observer's understanding of a phenomenon to be supernatural, such as lightning in days hence, does not rend it supernatural. However, if something is proposed by definition to be non-natural, then by definition it is impossible to investigate it as it supersedes natural boundaries and thus supersedes science which is confined to natural phenomena.
Look at it logically, so that you understand the distinction.
Person B proposes that a potential Event A *appears* to be supernatural.
The appearance of a potential Event A being supernatural to Person B does not make it so.
Thus, Phenomenon A is not necessarily supernatural.
Compare that with this:
PPerson B proposes that a potential Event A is *defined* as supernatural.
If potential Event A is defined to be supernatural, then by definition it is supernatural.
Thus, Phenomenon A is necessarily supernatural.
If you start off a scientific investigation by *stating* that your explanation is *by definition* supernatural, then *necessarily* your explanation is not testable.
But, that is the philosophical contention scientists take with ID. The scientific contention regards IC and CSI, the main predictions and *central* components of ID, both of which have been falsified. As these central pieces of evidence have been falsified, then *necessarily* ID is falsified.
If science were capable of testing methodological naturalism to see whether it is in accord with objective reality then we might know whether this viewpoint will yield knowledge in accord with the same. Regardless, methodological naturalism is capable of recognizing intelligent design. It may have difficulty ascertaining the disposition or personal nature of an intelligent designer, but it has little difficulty in ascertaining the presence of organized matter performing specific functions - a hallmark of intelligent design.