Yes, the protestants are wrong to accuse Catholics of praying to graven images and the like. When we pray BEFORE a crucifix or a statue we do not pray TO the images; we pray to God. I think that is what we must be clear about. We don’t pray to a crucifix but He whose image is on the crucifix.
Well, muslims have a field day claiming Catholics pray/worship to the cross. Then, when you ask why muslims pray/worship the Kaaba 5x aday. They quickly stand in defence & claim “direction”. There you go, you have a justification of the same.
That, too, is in "I don't care how you call it" category. I stand, or kneel, and pray before a Crucifix or a Holy Icon because the object became holy though the image it bears; I pray to the crucifix because in the English language not sullied by Protestant police station style of interrogations, it is most clear and economical way of describing what I do. If they want to know what is in my mind when I pray, well, they can ask and I will tell them; it is curious that they never ask that.
Once again, this difference in worship alone is sufficient reason not to hold any ecumenical prayers.