It depends on how valuable the information stored on it is, and thus, how determined and ruthless your potential attackers are likely to be. Give me physical control of the box, physical control of you, and a pair of pliers, and I'll be on as root in less than twenty minutes, I expect. And there are plenty of installations out there where this is a very real risk - don't underestimate the utility of access control for protecting you and the system. There's not much point in breaking your fingers to get your password if I still can't get at the machine once I have it.
On some very secure systems, the user is given two passwords. One password is for normal access, and the second password is for "duress" access.
Duress is when a foreign agent is beating the crap out of your wife and baby daughter until you give him a password that accesses all of your information.
Except that while the "duress" password will give said access to most or all of your critical data, it also signals your own team to come swooping in to the rescue.
In the civilian world, something very similar is done with web honeypot servers, which lure crackers in with "real looking" data and help the feds bust them.
Typically, a good honeypot server will have "valid" credit card numbers on it, for instance, but using those numbers will have the feds on you in mere minutes, as they are programmed in to the network to signal that a crime is in progress. Use the credit card number from a honeypot and you **will** get busted.