I don't mean to be dense, but how do we know that it is not the other way around?
BTW: Thanks for the interesting posts, I'm on my way out for a while.
Of course we should use whichever coordinate system seems most workable. For gravitational fields we have Riemannian space, and that seems handy enough. If time fields use the same kind of field, it would not matter as a practical thing. Whichever is convenient and attracts funding is the way to go.