Some people do have to worry. Ya can get castles pretty cheap[Europe]. Cheap is always subjective.
The cost of upkeep and stuff is likely prohibitive, those peasant workers want to be paid these days...
I am reminded of a Remington Steele episode in season 5 where he somehow inherits a castle and there are so many past due bills and other high bills he can’t do anything with it.