It’s basic property rights. They have likely had access for many years. That establishes an easement and they have the right to use it. If the government doesn’t like that they can always buy the house.
The Feds need to put that land up for sale.
We should not be in the business of providing vacation properties.