well, it’s not so hard to see why he went for this — firstly, the canon had not been put together, and just a cursory read of the OT and NT seems to suggest two vastly different gods — an OT God of fire and brimstone who punishes and calls for extinction of peoples and a NT God of love. If we read scripture in bits and pieces and out of context, we can easily fall for the same story that Marcion did.
The Marcionites according to history ran an ecclesiastical organization parallel to the Church of Rome for 300 years after which they were absorbed into Manicheism. The Manicheans were the sect that Augustine converted to Rome from and during his day they had a large community in Rome with which he was in contact trying to bring them also into the Church of Rome.