Even had Justice Wayne voted as a rebel sympathizer, unless half of the other Justices saw the law the same way he did it would have made no difference. Which FIVE justices do you think were rebel sympathizers?
The evidence is clear. The radicals could only have feared losing the vote of 5 of the 9 justices. Fearing that more than half of the justices would find their actions unlawful, they proceeded to play with the Court's membership to rig the decisions.
And when that failed, as in the 9-zip decision in Ex Parte Milligan, they interfered with the jurisdiction of the court as in McCardle.
McCardle was a newpaper editor in Vicksburg, Mississippi, who was arrested by federal officials for writing a series of newspaper articles that were highly critical of Reconstruction and especially of the military rule of the sourth following the Civil War....Constitutional Law, Principles and Policies, Second Edition, 2002, Erwin Chemerinsky,McCardle contended that the Military Reconstruction Act was unconstitutional in that it provided for military trials for civilians. He also claimed that his prosecution violated specific Bill of Rights provisions, including the First, Fifth, and Sixth Amendments. The United States government argued that the federal courts lacked jurisdiction to grant habeas corpus to McCardle under the 1867 Act. The federal government read the 1867 statute, despite its language to the contrary, as providing federal court relief only for state prisoners. The Supreme Court rejected this contention and set the case for argument on the merits of McCardle's claim that the Military Reconstruction Act and his prosecution were unconstitutional
On March 9, 1868, the Supreme Court held oral arguments on McCardle's constitutional claims. Three days later, on March 12, 1868, Congress adopted a rider to an inconsequential tax bill that repealed that part of the 1867 statute that authorized Supreme Court appellate review of writs of habeas corpus. Members of Congress stated that their purpose was to remove the McCardle case from the Supreme Court's docket and thus prevent the Court from potentially invalidating Reconstruction. Representative Wilson declared that the "amendment [repealing Supreme Court authority under the 1867 Act is] aimed at striking at a branch of the jurisdiction of the Supreme Court... thereby sweeping the [McCardle] case from the docket by taking away the jurisdiction of the Court."
On March 25, 1868, President Andrew Johnson vetoed the attempted repeal of Supreme Court jurisdiction. It should be noted that this was five days before the Senate was scheduled to begin its impeachment trial of President Johnson and that the grounds for impeachment focused solely on his alleged obstruction of Reconstruction. President Johnson declared: "I cannot give my assent to a measure which proposes to deprive any person restrained of his or her liberty in violation of the Constitution.... from the right of appeal to the highest judicial authority known to our government." The Congress immediately overrode President Johnson's veto on March 27, 1868.
The Military Reconstruction Act was unconstitutional. The radical Congress knew it was unconstitutional. They knew the Supreme Court would rule that it was unconstitutional. In order that they should be able to perform unconstitutional acts, they took away the jurisdiction of the Supreme Court to rule on the case.