Then why are Protestants castigated for wanting to return to it and use IT as the point of authority for spiritual matters?
You are not castigated for that -- I am doing the same thing in denouncing Protestantism as a counterscriptural invention. Your error is that you deviate from the scripture in matters central to the faith: you deny the guidance of the Church, the sacraments of the Church, and you invented the false theory of salvation being by faith alone. All that is contrary to the scripture. Further, while the scripture is indeed "a point of authority", your error is to consider it the sole point of authority, whereas, again, that is not in the scripture.