Maybe it’s the way I was brought up. Reading wasn’t a luxury, it was mandatory. My mother was a school teacher and we read books, LOTS of books. Perhaps if you think of it as a luxury then other mundane things will take up your time.
Books compete with other forms of “entertainment” such as TV, video games, movies, the internet, etc. They are time consuming and relatively pricey to own.