The Framers of the Constitution made explicit provision for this type of check in the Constitution itself. Article I, Section 8 and Article III, Sections 1 and 2 grant Congress the authority to establish inferior federal courts, determine their jurisdiction and make exceptions to the Supreme Court's appellate jurisdiction.
All of that is true, but you simply cannot get around a judicial review of laws passed by Congress.
Yes, Congress could establish an inferior court and limit appellate review by the USSC in the area of marriage. But the new court could then rule that a marriage law passed by Congress was unconstitutional and the USSC would not be able to overturn the ruling.
Keep in mind also that the new court's judges would still have to be appointed by the President and approved by Congress.
That's only true in cases where courts are needed to bring the act of Congress into effect. But the purpose of DOMA was to reduce the ability of the federal government to impose itself on the states. So Congress doesn't need federal courts at all to administer that law. All that's required is for states to exercise their freedom under DOMA.
It would be a very good day if and when a state defies a federal court that tries to unconstitutionally order it to do otherwise.