I doubt that the plan was to send the Blacks to Panama against their will or as slaves.
One of my ancestors was among the founders of the anti-slavery Republican party in the state of NY and had no such plans for “negroes”. Instead, he hired two to work on his property in NY, as free men.
I had another ancestor who actually did work on the Panama canal as a train engineer. He loved it.
You don't think so? Fine. But the historians would disagree, who've looked into this idea.
You don't get 2,000,000 or 1,000,000 or 500,000 of anybody to go to a fever-infested jungle by just asking them.