Stealing stuff from somebody's house is a very poor analogy in this case. A better comparison would be finding something of value in somebody's trash once they discarded it. The person that finds it, rescues it from the elements, preserves it, documents it, even profits from its display is under no obligation to return it to the person, people, nation or culture that discarded it and was willing to let it fall into disrepair or decay.
No cultural treasures I am aware are discarded by those people whose ancestors created them. If any were discarded damaged and destroyed they were done so by the outside perps - like Elgin and Bingham.