BTW, have you tried out Mint XfCE? It may be what you are looking for...
No, I have not tried Mint. There are quite a few who swear by it.
It is XFCE based from what I have heard.
The issue is the underlying assumptions as to OS. It is not necessary for function to have all the “stuff” that is in the Ubuntu and Debian OS.
Remove the stuff that is required by Gnome desktop and by Unity and it is a lot more efficient. Some of that is probably necessary, but not all of it.
I have thought about trying Mint. The only thing I don’t like about Xubuntu is the upgrade process. I tried to use it long ago and had a crash and had to really work to get it recovered. That is why I am probably going to re-install soon.