In any case, NX is a hardware function, not a feature of the operating system. Windows systems on processors that support NX enjoy exactly the same protection. In addition, it's hardly a universal fix for buffer overflows. All NX does is prevent stack pages from being executed - it does not prevent the stack-smasher from pointing to another location in memory and executing code located there. And it can't.
I didn't say it "Can't be done" I said it is much more difficult to do. Yes, it is a hardware feature... but Windows sometimes uses programs that seem to require executing things in the stack, so it the protection is not enabled by default. In OS X it is enabled by default.
In additin, any malicious code would have to be OS dependent... and the likelyhood of it being Mac code is small. As I said, on a Mac the app, in this case Excel, would just crash when the stack was smashed.
If the code cannot be executed, how can it point to anything? It won't execute. Here is some salient comments from 6/7/2004 from Techrepublic.com on the benefits of NX implementation in Windows XP SP2 of the NX capability on certain CPUs:
NX support is added to Windows XP. NX (no execute) will allow NX-enabled CPUs to mark certain areas of memory as non-executable; that is, any code pushed into those areas (perhaps by malware such as Blaster or other viruses) will just sit there, unable to run and therefore will be rendered harmless. This will harden the OS against the notorious buffer overrun threats. NX is currently only supported for AMD’s K8 and Intel’s Itanium processors, but 32- and 64-bit support for this important security feature is expected in most future processor releases... The NX protection mentioned above is an excellent example of something that is definitely a powerful improvement from the security standpoint. However, NX has already been reported to have caused considerable problems (at least in the 64-bit version).
In any case, the NX bit is not set to ON by default in Windows because only a few CPUs are NX capable... and some programs break on encountering non-executable stacks.
I may be wrong, but it is my understanding that OS X sets the NX bit for data pages as well and prevents code execution there as well. This particular Excel vulnerability is apparently an overflow in the data stack... not the program stack.
...it does not prevent the stack-smasher from pointing to another location in memory and executing code located there.
While a return address might redirect the instructions to a malicious app in another non-protected memory area, on a Mac how does the malicious code get loaded there??? Are we talking about a much more sophisticated attack, or a buffer overflow? To accomplish installing malicious code in an executable area raises the bar considerably on a Mac. It would also seem to be overkill to use a buffer overflow to redirect to a surrupticiously installed program... if you have already done that, it should be trivial to get it to run.
Of course, the real solution for either Windows or Mac users to any trojan Excel file is not to download it in the first place...