No, since in astronomy, there is something called the distance scale. It is a series of experiments and observations that use a variety of astronomical phoenomenon to build a reliable set of distances from sources as close as the nearest star to distant quasars and galaxies.
The Key Project of the Hubble Space Telescope was to minimise the amount of error in the distance scale and to use those measurements to determine the Hubble Constant to a good deal of precision.
The distance scale starts off with a fairly straightforward trigonometric relation called the Parallax Distances using parallax are rock solid. However, they aren't good to very far, only about 10,000 light years. However, given a constant speed of light, that age of the universe already gives contemporary creationist theory fits.
Next on the distance scale utilizes statistics, and basic observations of a large amount of stars to come up with a constant diagram (called the Hertzsprung-Russell Diagram) which shows that stars of a certain color and brightness tend to lay along a certain region of it. Give the spectrum of a star and its brightness in the sky, you can find the distance of it to about 20%.
From there, you can hook up with Cepheid variables. These variable stars have a period dependent on the mass of the star. Mass depends on absolute luminosity of the star, and from there you can determine a distance. This measurement to several nearby galaxies was the first relationship that the Hubble Key Project used to find the distances to several nearby galaxies.
To check their distances, they used a number of other distance measurements such as the Thully-Fisher relationship, Type-1a Supernovae, and other, more esoteric methods.
So you see, the concept of things at large distances (and because of the speed limit of light, large ages) is something that is not built on by a whim, but is an interconnected set of concepts and evidence. You can't just knock one down and expect it to fall like a house of cards. It is here to stay.