Slavery was obviously a large contributing factor to the Civil War, but it was hardly the only factor. The north and south were divided on a number of issues and the conflict might have occurred even if slaves never were part of the picture.
Grant didn't say that there weren't other factors, in fact he mentions having weighed them before coming to his conclusion.
I'm not trying to be confrontational, but to say that the civil war might have occurred without slavery being in the picture is a bit of a stretch. It kind of like saying someone might have gotten hit by lightning even if they weren't holding a lightning rod while standing in a mud puddle. Yes, it's statistically possible but not very probable.
A lot of work, compromises and several presidents devoted much of their time prior to the civil war in hopes of preventing it, in the end it always boiled down to the issue of slavery wich led inexorably to the war.