That's the reason they make kitty doors. They do well when they're allowed to be inside or outside at their own choice.
There isn't an animal on the face of the earth that evolved to live as a ward of humans in a confined space. If that were true of house cats, they would have lost their claws and excellent night vision long ago. They also wouldn't have retained their stalking/hunting instincts, but they have.
Our cats don't have any indoor provisions, but we let them in whenever they want. They stay in for as long as they like, and when they indicate they've had enough, we let them out. They're happy, contented, and well adjusted to that lifestyle, and so are we. I wouldn't change a thing.
After a lifetime of owning cats, I think you are totally right. Since I now have about 4 cats, I’m thinking about a kitty door.