If an EO is in direct violation of the Constitution or a previous SCOTUS ruling - it would be struck down, probably by a 9-0 ruling.
Lets assume that Obama issues an EO banning hi-cap magazines.
SCOTUS has ruled that citizens have the right to possess firearms [District of Columbia v. Heller]. But, it also left open that restrictions on it might be passed by legislative action. However, an EO is an executive action - not a legislative action.
Putting poolitical ideology aside, if SCOTUS were to allow such an EO to stand - it would be ceding all of its legal and moral authority to exist. They might as well pack up their toys and retire, for they would be stating that the Executive Department can supercede the Constitution, the Legislative Department, and the Judicial Department by fiat.
This scenario is a turf war and SCOTUS would assert its authority [probably 9-0] in order to preserve the constitutional concept of the system of Checks and Balances.
God only knows how traitor Chief Justice Roberts would interpret the law however.