It looks to me like you'd have the problem if you shared the server with an untrusted party. If you have the server all to yourself, that's different.
At many locations, I share the server with anywhere from 10 to 1200 users depending upon the size of the installation.