Some of the questions posed seem very revealing of the agenda:
Yudkowsky: How can we shape the intelligence explosion for the benefit of humanity?
Peterson: How can we safely bring humanity and the biosphere through the Singularity?
Kurzweil: Will the Singularity be a soft (gradual) or hard (rapid) take off and how will humans stay in control?
Doctorow: Will our technology serve us, or control us?
Consider who exactly is the "we" referred to in these questions. Is it possible that these questioners literally see themselves as the "we", the Controllers, the ones C.S. Lewis referred to as "The Conditioners" in his prescient little book The Abolition of Man? ("For the power of Man to make himself what he pleases means, as we have seen, the power of some men to make other men what they please.") Perhaps the questions might be more aptly put, "how will certain humans stay in control" of others?
"... In order to understand fully what Man's power over Nature, and therefore the power of some men over other men, really means, we must picture the race extended in time from the date of its emergence to that of its extinction. Each generation exercises power over its successors: and each, in so far as it modifies the environment bequeathed to it and rebels against tradition, resists and limits the power of its predecessors. This modifies the picture which is sometimes painted of a progressive emancipation from tradition and a progressive control of natural processes resulting in a continual increase of human power. In reality, of course, if any one age really attains, by eugenics and scientific education, the power to make its descendants what it pleases, all men who live after it are the patients of that power. They are weaker, not stronger: for though we may have put wonderful machines in their hands we have pre-ordained how they are to use them. And if, as is almost certain, the age which had thus attained maximum power over posterity were also the age most emancipated from tradition, it would be engaged in reducing the power of its predecessors almost as drastically as that of its successors. And we must also remember that, quite apart from this, the later a generation comes—the nearer it lives to that date at which the species becomes extinct—the less power it will have in the forward direction, because its subjects will be so few. There is therefore no question of a power vested in the race as a whole steadily growing as long as the race survives. The last men, far from being the heirs of power, will be of all men most subject to the dead hand of the great planners and conditioners and will themselves exercise least power upon the future.The real picture is that of one dominant age—let us suppose the hundredth century A.D.—which resists all previous ages most successfully and dominates all subsequent ages most irresistibly, and thus is the real master of the human species. But then within this master generation (itself an infinitesimal minority of the species) the power will be exercised by a minority smaller still. Man's conquest of Nature, if the dreams of some scientific planners are realized, means the rule of a few hundreds of men over billions upon billions of men. There neither is nor can be any simple increase of power on Man's side. Each new power won by man is a power over man as well. Each advance leaves him weaker as well aas stronger. In every victory, besides being the general who triumphs, he is also the prisoner who follows the triumphal car."
C.S. Lewis
Cordially,
"Machine learning" is a reasonable term for the process they are trying to perfect. Again, can a machine "know that it exists" in the same sense as a human? Who can even say any human other than ourselves is self-aware and not an automaton? We assume it's true on the basis of outward actions and responses.
I know a lot of Christians think C.S. Lewis is some awesome philosopher, but as this example shows, I think not.
Cheers!
(...or read up Lord Feverstone's recruitment speech to Mark Studdock in That Hideous Strength)
Cheers!