Well, then, on that basis I guess you'd have to toss out any conception of morality that does not follow the empirically observable facts of nature.
Of course, that would only leave us with relativist moralities like Utilitarianism, Social Darwinism, and Might Makes Right -- but hey, what the heck, it's scientifically based, right?
As it happens, religion is an exceedingly good basis for certain types of argument. What's an exceedingly bad argument is to take your approach, and to irrationally rule out the possible existence of God.