I agree with you that as a foreigner living in a different culture, you learn to adapt. You are not going to change the culture. Those who learn that lesson enjoy life more than those who struggle to get the culture to adapt to them. If it becomes so oppressive, you should leave.
I lived in Saudi Arabia for five years. My wife couldn't drive and had to wear conservative dress or an abaya. We had to eat in segregated restaurants and wait outside closed stores during prayer call. Still, we enjoyed our stay and the experience. Those who were upset with these conditions hated their time in the Kingdom. That said, adapting doesn't mean that you cannot criticize such practices or find them abhorrent.
Much of this is a matter of one's own mind. Maybe it makes Americans and others feel better to tag some other people as racist; and maybe 'they' are or are not. If it makes one feel good to do this, maybe that's fine, too. But the feelings of triumphalism and superiority may not last for long.
It has nothing to do with making one feel good with "feelings of triumphalism and superiority." Rather, it has to do with absolute moral values versus moral or ethical relativism. Racism is wrong in Japan and much as it is in the US. It doesn't matter if the racism is enshrined in 2000 years of culture or not. The Japanese should not be immune from criticism anymore than we are. Denial just ain't a river in Egypt.
Fine. Change it then. Good luck.