Does Mac have any listeners running in a default install?
If there's no way to connect, hackers will have to try to trick you into installing their malware.
I suspect that most OS X security flaws involve the hacker gaining physical access to the computer, or for the user to install a trojan of some kind.
In terms of personal computing, if the hacker has physical access to your computer you have other more serious problems.