I didn't say it "Can't be done" I said it is much more difficult to do. Yes, it is a hardware feature... but Windows sometimes uses programs that seem to require executing things in the stack, so it the protection is not enabled by default. In OS X it is enabled by default.
In additin, any malicious code would have to be OS dependent... and the likelyhood of it being Mac code is small. As I said, on a Mac the app, in this case Excel, would just crash when the stack was smashed.
If the code cannot be executed, how can it point to anything? It won't execute. Here is some salient comments from 6/7/2004 from Techrepublic.com on the benefits of NX implementation in Windows XP SP2 of the NX capability on certain CPUs:
NX support is added to Windows XP. NX (no execute) will allow NX-enabled CPUs to mark certain areas of memory as non-executable; that is, any code pushed into those areas (perhaps by malware such as Blaster or other viruses) will just sit there, unable to run and therefore will be rendered harmless. This will harden the OS against the notorious buffer overrun threats. NX is currently only supported for AMD’s K8 and Intel’s Itanium processors, but 32- and 64-bit support for this important security feature is expected in most future processor releases... The NX protection mentioned above is an excellent example of something that is definitely a powerful improvement from the security standpoint. However, NX has already been reported to have caused considerable problems (at least in the 64-bit version).
In any case, the NX bit is not set to ON by default in Windows because only a few CPUs are NX capable... and some programs break on encountering non-executable stacks.
I may be wrong, but it is my understanding that OS X sets the NX bit for data pages as well and prevents code execution there as well. This particular Excel vulnerability is apparently an overflow in the data stack... not the program stack.
...it does not prevent the stack-smasher from pointing to another location in memory and executing code located there.
While a return address might redirect the instructions to a malicious app in another non-protected memory area, on a Mac how does the malicious code get loaded there??? Are we talking about a much more sophisticated attack, or a buffer overflow? To accomplish installing malicious code in an executable area raises the bar considerably on a Mac. It would also seem to be overkill to use a buffer overflow to redirect to a surrupticiously installed program... if you have already done that, it should be trivial to get it to run.
Of course, the real solution for either Windows or Mac users to any trojan Excel file is not to download it in the first place...
However, in the more invidious application, the data filling and overflowing the buffer is carefully constructed to be a series of opcodes that will land on the stack, and the return address will be overwritten to point to this malicious set of opcodes, and then you're in a world of hurt. NX-protection prevents this by making the stack pages non-executable - you can overwrite the return address, but you can't pass execution on to your malicious payload.
HOWEVER, NX-protection cannot prevent the return address from pointing to some other location in memory, and passing execution off to that location - in particular, you can pass execution off to some location containing convenient OS opcodes, which you can then use to further your nefarious scheme. This is the whole rationale behind the Metasploit project, to document the locations of Windows opcodes in memory, so that they can be exploited in this manner. But it's not unique to Windows - if you know the location of vulnerable opcodes in OS X, you can call them from a stack exploit just as easily as in Windows, and NX can't prevent it, because it's passing execution off the stack.