Why then, did Delaware not join the secession? Surely they would have if slavery was the reason. My point still stands.
Delaware would have been part of the "second wave" of seceding states (e.g. Virginia or Tennessee), which reacted after the Confederacy began the war by joining the new nation. Since at that time neighboring Maryland had already been occupied by federal soldiers, there must have been a strong sense amongst the pro-slavery faction in Delaware that their position would be strategically hopeless. Furthermore, there was obviously a strong pro-Union faction in the tiny state, as it sent nine regiments to the federal forces.