A lot of Speer's history is self-serving, so you have to take what he says there and elsewhere guardedly. What he says about Hitler's disastrous interference in production of the Messerschmitt Me 262 is confirmed by several other sources.
Note the good analogy you raise for another example of fascist economics (Corporatism); smart people warned that Obamacare would be a complete disaster, but the big insurance companies readily signed off on it, just like Porsche with the massive tanks, Messerschmitt with the bomber design changes to the Me-262, etc.
If Hitler had put Albert Speer in charge of armaments production from the start, and had left him alone to do his job, the allies would have have faced a much more uphill struggle to destroy the Third Reich.