The two are inseparable. The larger the objective lens, the more light. The limitation is in restricting the amount of light that can be passed to the lens as in a hood or tube extension prior to the objective. You gain a larger FOV by increasing the degree of arc to the lens, that way you can reduce the tube diameter, but with this you run the risk of increased vignetting or distortion.
... and amount of eye relief, and amount of reticle adjustment in MOA, and variable power, and parallax at different distances, etc.
Generally speaking, for two rifle scopes with the same quality of glass, and with the same magnification, the difference between a larger objective lens and the smaller objective lens is not brightness of image but field of view.