Yes, I'm rather sure they would have kept them. As spoils of war, their change in ownership can be justified.
As for "allowed", I'm not really sure who would be the one giving Germany "permission" if they had just conquered the planet. In international relations, "might makes right". The fact that, as the current world leader, America does its best to ensure that principles of justice are follwed (and the rest of the planet gladly and happily accepts... and occasionally takes advantage of... our very wise lead) does NOT mean that this is the only system that has existed or could be accepted. Go back in time, find Alexander the Great and Genghis Khan, and tell them that every item of wealth, stature, food, medical supplies, and strategic resource should be returned. See what they say. Then see if you can convince them otherwise with arguments for principles of justness and fairness. Talk to any citizen about this "injustice", and their likely response will be "Yeah, but what are ya gonna do about it?"
We're very fortunate (spoiled, really) to live in a super-power that is potentially responsive to the complaints of everyone in the world. However, that doesn't mean we should undo every transaction throughout history that someone has a grudge against.
(Interesting counter-point for anyone to address: Why can't Britain claim that their cultural heritage includes the spoils they claimed throughout the planet? Their cultural peak was during this era of Imperialism when "the sun never set on the British Empire". Taking the Stone away would harm the British historical sense of self!!!)