US soldiers took those flags after great personal danger, prying them from the hands of Japanese soldiers they had just killed.
To give a flag back, even 70 years later, speaks to the character of those who do it.
As to your situation, yes that flag would have been a great piece of family history for you. I would have kept it too.
I have a sword that my dad got from a dead Japanese Officer during the Battle for Manila. Give it back not on your life!