The following words of Stevens are consistent with Taylor's version (Stevens' speech of Dec 18, 1865):
They [Southern States] ought never to be recognized as capable of acting in the Union, or of being counted as valid States, until the Constitution shall have been so amended as to make it what its framers intended; and so as to secure perpetual ascendency to the party of the Union; and so as to render our republican Government firm and stable forever. The first of those amendments is to change the basis of representation among the States from Federal numbers to actual voters. . . . With the basis unchanged the 83 Southern members, with the Democrats that will in the best times be elected from the North, will always give a majority in Congress and in the Electoral college. . . . I need not depict the ruin that would follow. . ....This Congress is bound to provide for them [freed slaves] until they can take care of themselves. If we do not furnish them with homesteads, and hedge them around with protective laws; if we leave them to the legislation of their late masters, we had better have left them in bondage.
Other sources note that Stevens wanted to break up large landholdings and give the land to former slaves, so that position of Stevens seems consistent with Taylor's version of the conversation.
Finally, before the war the Radical Republicans held that there were higher laws than the Constitution. Given that, the disdain with which Stevens reportedly characterized the Constitution to Taylor is not all that surprising either.