Actually, his evidence was stronger than that. Certainly, the visible evidence of the phases of Venus was an extremely powerful confirmation of part of the Copernicus model. That evidence made it undeniable that Venus orbited the sun. But, as you say, that didn't show anything about the earth's movement. Specifically, the phases of Venus didn't rule out hybrid models that allowed for the planets to orbit the sun, while the sun and everything else still orbited the earth. There were such models at the time.
Galileo's observations, however, took away yet another argument for a stationary earth -- one which has been largely forgotten. It had been argued that the earth must be fixed in place because if it moved, it would leave the moon behind! That sounds goofy now, but Galileo flourished a generation before Isaac Newton, and in Galileo's day, no one realized that gravity held the moon in its orbit around the earth.
What Galileo argued here was a deduction that followed from his discovery that Jupiter had moons. Those moons clearly orbited Jupiter, and somehow they didn't get left behind, even though it was obvious to all that Jupiter was moving. Thus, although then inexplicable, our moon's similar behavior couldn't be advanced as "proof" that the earth was stationary. Therefore, Galileo's work left no argument remaining that the earth was stationary -- except the then-current interpretation of scripture.
Galileo was very good at tearing down other theories, but was poor at proving his own. So the Church was stuck in a do-or-die instigated by Galileo of defending an old model whose evidence was debunked, or supporting a new system whose evidence was scant. That would somewhat be akin to asking you to embrace Intelligent Design if all the evidence for the theory of evolution was adequately debunked(I'm not saying anything pro or contra about these two, it's just an example). It's something I don't think you would do. The logical thing to do would be to hold one's opinion until there were enough facts to make a prudent judgement. Galileo demanded nothing of the sort, and demanded the scripture be reinterpreted immediately. Hence his problems.